PyTorch torchvision - Docs
https://pytorch.org/vision/main/index.html

1. torchvision module code

https://pytorch.org/vision/main/_modules/index.html

torchvision.datasets._optical_flow
torchvision.datasets._stereo_matching
torchvision.datasets.caltech
torchvision.datasets.celeba
torchvision.datasets.cifar
torchvision.datasets.cityscapes
torchvision.datasets.coco
torchvision.datasets.country211
torchvision.datasets.dtd
torchvision.datasets.eurosat
torchvision.datasets.fakedata
torchvision.datasets.fer2013
torchvision.datasets.fgvc_aircraft
torchvision.datasets.flickr
torchvision.datasets.flowers102
torchvision.datasets.folder
torchvision.datasets.food101
torchvision.datasets.gtsrb
torchvision.datasets.hmdb51
torchvision.datasets.imagenet
torchvision.datasets.imagenette
torchvision.datasets.inaturalist
torchvision.datasets.kinetics
torchvision.datasets.kitti
torchvision.datasets.lfw
torchvision.datasets.lsun
torchvision.datasets.mnist
torchvision.datasets.moving_mnist
torchvision.datasets.omniglot
torchvision.datasets.oxford_iiit_pet
torchvision.datasets.pcam
torchvision.datasets.phototour
torchvision.datasets.places365
torchvision.datasets.rendered_sst2
torchvision.datasets.sbd
torchvision.datasets.sbu
torchvision.datasets.semeion
torchvision.datasets.stanford_cars
torchvision.datasets.stl10
torchvision.datasets.sun397
torchvision.datasets.svhn
torchvision.datasets.ucf101
torchvision.datasets.usps
torchvision.datasets.vision
torchvision.datasets.voc
torchvision.datasets.widerface
torchvision.io.image
torchvision.io.video
torchvision.io.video_reader
torchvision.models._api
torchvision.models.alexnet
torchvision.models.convnext
torchvision.models.densenet
torchvision.models.detection.faster_rcnn
torchvision.models.detection.fcos
torchvision.models.detection.keypoint_rcnn
torchvision.models.detection.mask_rcnn
torchvision.models.detection.retinanet
torchvision.models.detection.ssd
torchvision.models.detection.ssdlite
torchvision.models.efficientnet
torchvision.models.feature_extraction
torchvision.models.googlenet
torchvision.models.inception
torchvision.models.maxvit
torchvision.models.mnasnet
torchvision.models.mobilenetv2
torchvision.models.mobilenetv3
torchvision.models.optical_flow.raft
torchvision.models.quantization.googlenet
torchvision.models.quantization.inception
torchvision.models.quantization.mobilenetv2
torchvision.models.quantization.mobilenetv3
torchvision.models.quantization.resnet
torchvision.models.quantization.shufflenetv2
torchvision.models.regnet
torchvision.models.resnet
torchvision.models.segmentation.deeplabv3
torchvision.models.segmentation.fcn
torchvision.models.segmentation.lraspp
torchvision.models.shufflenetv2
torchvision.models.squeezenet
torchvision.models.swin_transformer
torchvision.models.vgg
torchvision.models.video.mvit
torchvision.models.video.resnet
torchvision.models.video.s3d
torchvision.models.video.swin_transformer
torchvision.models.vision_transformer
torchvision.ops.boxes
torchvision.ops.ciou_loss
torchvision.ops.deform_conv
torchvision.ops.diou_loss
torchvision.ops.drop_block
torchvision.ops.feature_pyramid_network
torchvision.ops.focal_loss
torchvision.ops.giou_loss
torchvision.ops.misc
torchvision.ops.poolers
torchvision.ops.ps_roi_align
torchvision.ops.ps_roi_pool
torchvision.ops.roi_align
torchvision.ops.roi_pool
torchvision.ops.stochastic_depth
torchvision.transforms.autoaugment
torchvision.transforms.functional
torchvision.transforms.transforms
torchvision.transforms.v2._augment
torchvision.transforms.v2._auto_augment
torchvision.transforms.v2._color
torchvision.transforms.v2._container
torchvision.transforms.v2._deprecated
torchvision.transforms.v2._geometry
torchvision.transforms.v2._meta
torchvision.transforms.v2._misc
torchvision.transforms.v2._temporal
torchvision.transforms.v2._type_conversion
torchvision.transforms.v2.functional._augment
torchvision.transforms.v2.functional._color
torchvision.transforms.v2.functional._deprecated
torchvision.transforms.v2.functional._geometry
torchvision.transforms.v2.functional._meta
torchvision.transforms.v2.functional._misc
torchvision.transforms.v2.functional._temporal
torchvision.transforms.v2.functional._type_conversion
torchvision.transforms.v2.functional._utils
torchvision.tv_tensors
torchvision.tv_tensors._bounding_boxes
torchvision.tv_tensors._dataset_wrapper
torchvision.tv_tensors._image
torchvision.tv_tensors._mask
torchvision.tv_tensors._torch_function_helpers
torchvision.tv_tensors._tv_tensor
torchvision.tv_tensors._video
torchvision.utils

2. torchvision.models.mobilenetv3

https://pytorch.org/vision/main/_modules/torchvision/models/mobilenetv3.html

activation_layer = nn.Hardswish if cnf.use_hs else nn.ReLU

Searching for MobileNetV3
https://arxiv.org/abs/1905.02244

torchvision/models/mobilenetv3.py
https://github.com/pytorch/vision/blob/main/torchvision/models/mobilenetv3.py

Figure 6. Sigmoid and swish nonlinearities and ther “hard” counterparts.

We replace sigmoid function with its piece-wise linear hard analog: $\frac{\text{ReLU6}(x+3)}{6}$ similar to [11, 44]. The minor difference is we use ReLU6 rather than a custom clipping constant. Similarly, the hard version of swish becomes

$$h-\text{swish}[x]=x\frac{\text{ReLU6}(x+3)}{6}$$

A similar version of hard-swish was also recently proposed in [2]. The comparison of the soft and hard version of sigmoid and swish nonlinearities is shown in figure 6. Our choice of constants was motivated by simplicity and being a good match to the original smooth version.

In our experiments, we found hard-version of all these functions to have no discernible difference in accuracy, but multiple advantages from a deployment perspective. First, optimized implementations of ReLU6 are available on virtually all software and hardware frameworks. Second, in quantized mode, it eliminates potential numerical precision loss caused by different implementations of the approximate sigmoid. Finally, in practice, h-swish can be implemented as a piece-wise function to reduce the number of memory accesses driving the latency cost down substantially.

3. MobileNet V3

https://pytorch.org/vision/main/models/mobilenetv3.html

All the model builders internally rely on the torchvision.models.mobilenetv3.MobileNetV3 base class.

3.1. mobilenet_v3_large

https://pytorch.org/vision/main/models/generated/torchvision.models.mobilenet_v3_large.html

Constructs a large MobileNetV3 architecture from Searching for MobileNetV3.

3.2. mobilenet_v3_small

https://pytorch.org/vision/main/models/generated/torchvision.models.mobilenet_v3_small.html

Constructs a small MobileNetV3 architecture from Searching for MobileNetV3.

4. PyTorch Hardsigmoid

https://pytorch.org/docs/stable/generated/torch.nn.Hardsigmoid.html

Applies the Hardsigmoid function element-wise.

Hardsigmoid is defined as:

$$\text{Hardsigmoid}(x)=\{\begin{array}{ll}0& \text{if }x\le -3,\\ 1& \text{if }x\ge +3,\\ x/6+1/2=x/6+3/6=(x+3)/6& \text{otherwise}\end{array}$$

Args:

inplace (bool) - can optionally do the operation in-place. Default: False

Shape:

• Input: (*), where * means any number of dimensions.
• Output: (*), same shape as the input.

Examples:

>>> m = nn.Hardsigmoid()
>>> input = torch.randn(2)
>>> output = m(input)

4.1. MobileNetV3-Pytorch

MobileNetV3-Pytorch
https://github.com/leaderj1001/MobileNetV3-Pytorch/tree/master

model.py
https://github.com/leaderj1001/MobileNetV3-Pytorch/blob/master/model.py

class h_sigmoid(nn.Module):
    def __init__(self, inplace=True):
        super(h_sigmoid, self).__init__()
        self.inplace = inplace

    def forward(self, x):
        return F.relu6(x + 3., inplace=self.inplace) / 6.


class h_swish(nn.Module):
    def __init__(self, inplace=True):
        super(h_swish, self).__init__()
        self.inplace = inplace

    def forward(self, x):
        out = F.relu6(x + 3., self.inplace) / 6.
        return out * x

4.2. pytorch-mobilenet-v3

pytorch-mobilenet-v3
https://github.com/kuan-wang/pytorch-mobilenet-v3

mobilenetv3.py
https://github.com/kuan-wang/pytorch-mobilenet-v3/blob/master/mobilenetv3.py

class Hswish(nn.Module):
    def __init__(self, inplace=True):
        super(Hswish, self).__init__()
        self.inplace = inplace

    def forward(self, x):
        return x * F.relu6(x + 3., inplace=self.inplace) / 6.


class Hsigmoid(nn.Module):
    def __init__(self, inplace=True):
        super(Hsigmoid, self).__init__()
        self.inplace = inplace

    def forward(self, x):
        return F.relu6(x + 3., inplace=self.inplace) / 6.

4.3. mobilenetv3

mobilenetv3
https://github.com/xiaolai-sqlai/mobilenetv3

mobilenetv3.py
https://github.com/xiaolai-sqlai/mobilenetv3/blob/master/mobilenetv3.py

class hswish(nn.Module):
    def forward(self, x):
        out = x * F.relu6(x + 3, inplace=True) / 6
        return out


class hsigmoid(nn.Module):
    def forward(self, x):
        out = F.relu6(x + 3, inplace=True) / 6
        return out

5. TensorFlow HardSigmoid

5.1. mobilenetv3-tensorflow

mobilenetv3-tensorflow
https://github.com/Bisonai/mobilenetv3-tensorflow

layers.py
https://github.com/Bisonai/mobilenetv3-tensorflow/blob/master/layers.py

class ReLU6(tf.keras.layers.Layer):
    def __init__(self):
        super().__init__(name="ReLU6")
        self.relu6 = tf.keras.layers.ReLU(max_value=6, name="ReLU6")

    def call(self, input):
        return self.relu6(input)


class HardSigmoid(tf.keras.layers.Layer):
    def __init__(self):
        super().__init__(name="HardSigmoid")
        self.relu6 = ReLU6()

    def call(self, input):
        return self.relu6(input + 3.0) / 6.0


class HardSwish(tf.keras.layers.Layer):
    def __init__(self, name="HardSwish"):
        super().__init__(name=name)
        self.hard_sigmoid = HardSigmoid()

    def call(self, input):
        return input * self.hard_sigmoid(input)

5.2. MobilenetV3-Tensorflow

MobilenetV3-Tensorflow
https://github.com/frotms/MobilenetV3-Tensorflow

mobilenet_v3.py
https://github.com/frotms/MobilenetV3-Tensorflow/blob/master/mobilenet_v3.py

def relu6(x, name='relu6'):
    return tf.nn.relu6(x, name)


def hard_swish(x, name='hard_swish'):
    with tf.variable_scope(name):
        h_swish = x * tf.nn.relu6(x + 3) / 6
    return h_swish


def hard_sigmoid(x, name='hard_sigmoid'):
    with tf.variable_scope(name):
        h_sigmoid = tf.nn.relu6(x + 3) / 6
    return h_sigmoid

6. ONNX Operator Schemas - HardSigmoid

ONNX (Open Neural Network Exchange) is an open ecosystem for interoperable AI models.

docs/Operators.md
https://github.com/onnx/onnx/blob/main/docs/Operators.md

6.1. HardSigmoid

HardSigmoid takes one input data (Tensor) and produces one output data (Tensor) where the HardSigmoid function, $y=\text{max}(0,\text{min}(1,alpha\ast x+beta))$, is applied to the tensor elementwise.

• Attributes

alpha: float (default is 0.2)
Value of alpha.

beta: float (default is 0.5)
Value of beta.

• Inputs

X (differentiable): T
Input tensor

• Outputs

Y (differentiable): T
Output tensor

• Type Constraints

T: tensor(bfloat16), tensor(float16), tensor(float), tensor(double)
Constrain input and output types to float tensors.

• Examples

hardsigmoid

node = onnx.helper.make_node(
    "HardSigmoid", inputs=["x"], outputs=["y"], alpha=0.5, beta=0.6
)

x = np.array([-1, 0, 1]).astype(np.float32)
y = np.clip(x * 0.5 + 0.6, 0, 1)  # expected output [0.1, 0.6, 1.]
expect(node, inputs=[x], outputs=[y], name="test_hardsigmoid_example")

x = np.random.randn(3, 4, 5).astype(np.float32)
y = np.clip(x * 0.5 + 0.6, 0, 1)
expect(node, inputs=[x], outputs=[y], name="test_hardsigmoid")

hardsigmoid_default

default_alpha = 0.2
default_beta = 0.5
node = onnx.helper.make_node(
    "HardSigmoid",
    inputs=["x"],
    outputs=["y"],
)
x = np.random.randn(3, 4, 5).astype(np.float32)
y = np.clip(x * default_alpha + default_beta, 0, 1)
expect(node, inputs=[x], outputs=[y], name="test_hardsigmoid_default")

7. torch.nn.functional.hardsigmoid

https://www.paddlepaddle.org.cn/documentation/docs/en/guides/model_convert/convert_from_pytorch/api_difference/functional/torch.nn.functional.hardsigmoid.html

• torch.nn.functional.hardsigmoid

torch.nn.functional.hardsigmoid(input, inplace=False)

• paddle.nn.functional.hardsigmoid

paddle.nn.functional.hardsigmoid(x, slope=0.1666667, offset=0.5, name=None)

其中 PyTorch 和 Paddle 功能一致，仅 paddle 参数更多。

• 参数映射

PyTorch	PaddlePaddle	备注
input	x	输入 Tensor，仅参数名不一致。
-	slope	hardsigmoid 的斜率，PyTorch 无此参数，Paddle 保持默认即可。
-	offset	hardsigmoid 的截距，PyTorch 无此参数，Paddle 保持默认即可。
inplace	-	表示在不更改变量的内存地址的情况下，直接修改变量的值，Paddle 无此参数，一般对网络训练结果影响不大，可直接删除。

References

[1] Yongqiang Cheng, https://yongqiang.blog.csdn.net/
[2] module code, https://pytorch.org/vision/main/_modules/index.html