【BigData 2025】OrthoNets：正交通道注意力网络

---

本文提出OrthoNets正交通道注意力网络，认为FcaNet中DCT成功源于正交核滤波器。其简化空间压缩，用多个正交核滤波器，再进行类似SE的操作。在CIFAR-10上与ResNet-18对比实验，OrthoNet-18验证精度0.9406，参数11,270,602，性能相当，表明正交核对空间压缩有效。

☞☞☞AI 智能聊天, 问答助手, AI 智能搜索, 免费无限量使用 DeepSeek R1 模型☜☜☜

OrthoNets：正交通道注意力网络

摘要

        设计一种有效的通道注意机制要求人们找到一种用于最佳特征表示的有损压缩方法。尽管该领域最近取得了进展，但它仍然是一个悬而未决的问题。 FcaNet 是当前最先进的通道注意力机制，尝试使用离散余弦变换（DCT）找到这种信息丰富的压缩。 FcaNet 的一个缺点是 DCT 频率没有自然选择。为了解决这个问题，FcaNet 在 ImageNet 上进行了实验以找到最佳频率。我们假设频率的选择仅起辅助作用，而注意力过滤器有效性的主要驱动力是 DCT 内核的正交性。为了检验这个假设，我们使用随机初始化的正交滤波器构建了一个注意力机制。将此机制集成到 ResNet 中，我们创建了 OrthoNet。我们在 Birds、MS-COCO 和 Places356 上将 OrthoNet 与 FcaNet（和其他注意力机制）进行比较，并显示出优越的性能。在 ImageNet 数据集上，我们的方法可以与当前最先进的方法竞争或超越。我们的结果表明，滤波器的最佳选择是难以捉摸的，但是可以通过足够大量的正交滤波器来实现泛化。我们进一步研究了实施通道注意力的其他一般原则，例如其在网络中的位置和通道分组。

1. OrthoNets

        本文通过分析FcaNet中的DCT进行空间压缩的成功可能归功于正交核滤波器，因此本文简化空间压缩方法，仅使用多个正交核滤波器来进行空间压缩，然后使用于SE一样的操作：

$${\mathbf{F}}_{\text{ortho }}(X{)}_c=\sum _{h=1}^H\sum _{w=1}^W{K}_{c,h,w}{X}_{c,h,w}$$Fortho (X)c=h=1∑Hw=1∑WKc,h,wXc,h,w

       

2. 代码复现

2.1 下载并导入所需要的包

In [ ]

%matplotlib inlineimport paddleimport numpy as npimport matplotlib.pyplot as pltfrom paddle.vision.datasets import Cifar10from paddle.vision.transforms import Transposefrom paddle.io import Dataset, DataLoaderfrom paddle import nnimport paddle.nn.functional as Fimport paddle.vision.transforms as transformsimport osimport matplotlib.pyplot as pltfrom matplotlib.pyplot import figurefrom paddle import ParamAttrfrom paddle.nn.layer.norm import _BatchNormBaseimport mathfrom OrthoNets import *

   

2.2 创建数据集

In [3]

train_tfm = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.ColorJitter(brightness=0.2,contrast=0.2, saturation=0.2),
    transforms.RandomHorizontalFlip(0.5),
    transforms.RandomRotation(20),
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
])

test_tfm = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
])

    In [4]

paddle.vision.set_image_backend('cv2')# 使用Cifar10数据集train_dataset = Cifar10(data_file='data/data152754/cifar-10-python.tar.gz', mode='train', transform = train_tfm, )
val_dataset = Cifar10(data_file='data/data152754/cifar-10-python.tar.gz', mode='test',transform = test_tfm)print("train_dataset: %d" % len(train_dataset))print("val_dataset: %d" % len(val_dataset))

       

train_dataset: 50000
val_dataset: 10000

        In [5]

batch_size=256

    In [6]

train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=4)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False, drop_last=False, num_workers=4)

   

2.3 标签平滑

In [7]

class LabelSmoothingCrossEntropy(nn.Layer):
    def __init__(self, smoothing=0.1):
        super().__init__()
        self.smoothing = smoothing    def forward(self, pred, target):

        confidence = 1. - self.smoothing
        log_probs = F.log_softmax(pred, axis=-1)
        idx = paddle.stack([paddle.arange(log_probs.shape[0]), target], axis=1)
        nll_loss = paddle.gather_nd(-log_probs, index=idx)
        smooth_loss = paddle.mean(-log_probs, axis=-1)
        loss = confidence * nll_loss + self.smoothing * smooth_loss        return loss.mean()

   

2.4 OrthoNets

In [ ]

model = orthonet18(n_classes=10)
paddle.summary(model, (1, 3, 32, 32))

   

       

2.5 训练

In [9]

learning_rate = 0.1n_epochs = 100paddle.seed(42)
np.random.seed(42)

    In [ ]

work_path = 'work/model'model = orthonet18(n_classes=10)

criterion = LabelSmoothingCrossEntropy()

scheduler = paddle.optimizer.lr.MultiStepDecay(learning_rate, milestones=[30, 60, 90])
optimizer = paddle.optimizer.Momentum(parameters=model.parameters(), learning_rate=scheduler, weight_decay=5e-4)

gate = 0.0threshold = 0.0best_acc = 0.0val_acc = 0.0loss_record = {'train': {'loss': [], 'iter': []}, 'val': {'loss': [], 'iter': []}}   # for recording lossacc_record = {'train': {'acc': [], 'iter': []}, 'val': {'acc': [], 'iter': []}}      # for recording accuracyloss_iter = 0acc_iter = 0for epoch in range(n_epochs):    # ---------- Training ----------
    model.train()
    train_num = 0.0
    train_loss = 0.0

    val_num = 0.0
    val_loss = 0.0
    accuracy_manager = paddle.metric.Accuracy()
    val_accuracy_manager = paddle.metric.Accuracy()    print("#===epoch: {}, lr={:.10f}===#".format(epoch, optimizer.get_lr()))    for batch_id, data in enumerate(train_loader):
        x_data, y_data = data
        labels = paddle.unsqueeze(y_data, axis=1)

        logits = model(x_data)

        loss = criterion(logits, y_data)

        acc = accuracy_manager.compute(logits, labels)
        accuracy_manager.update(acc)        if batch_id % 10 == 0:
            loss_record['train']['loss'].append(loss.numpy())
            loss_record['train']['iter'].append(loss_iter)
            loss_iter += 1

        loss.backward()

        optimizer.step()
        optimizer.clear_grad()
        
        train_loss += loss
        train_num += len(y_data)
    scheduler.step()

    total_train_loss = (train_loss / train_num) * batch_size
    train_acc = accuracy_manager.accumulate()
    acc_record['train']['acc'].append(train_acc)
    acc_record['train']['iter'].append(acc_iter)
    acc_iter += 1
    # Print the information.
    print("#===epoch: {}, train loss is: {}, train acc is: {:2.2f}%===#".format(epoch, total_train_loss.numpy(), train_acc*100))    # ---------- Validation ----------
    model.eval()    for batch_id, data in enumerate(val_loader):

        x_data, y_data = data
        labels = paddle.unsqueeze(y_data, axis=1)        with paddle.no_grad():
          logits = model(x_data)

        loss = criterion(logits, y_data)

        acc = val_accuracy_manager.compute(logits, labels)
        val_accuracy_manager.update(acc)

        val_loss += loss
        val_num += len(y_data)

    total_val_loss = (val_loss / val_num) * batch_size
    loss_record['val']['loss'].append(total_val_loss.numpy())
    loss_record['val']['iter'].append(loss_iter)
    val_acc = val_accuracy_manager.accumulate()
    acc_record['val']['acc'].append(val_acc)
    acc_record['val']['iter'].append(acc_iter)    
    print("#===epoch: {}, val loss is: {}, val acc is: {:2.2f}%===#".format(epoch, total_val_loss.numpy(), val_acc*100))    # ===================save====================
    if val_acc > best_acc:
        best_acc = val_acc
        paddle.save(model.state_dict(), os.path.join(work_path, 'best_model.pdparams'))
        paddle.save(optimizer.state_dict(), os.path.join(work_path, 'best_optimizer.pdopt'))print(best_acc)
paddle.save(model.state_dict(), os.path.join(work_path, 'final_model.pdparams'))
paddle.save(optimizer.state_dict(), os.path.join(work_path, 'final_optimizer.pdopt'))

   

       

2.6 实验结果

In [11]

def plot_learning_curve(record, title='loss', ylabel='CE Loss'):
    ''' Plot learning curve of your CNN '''
    maxtrain = max(map(float, record['train'][title]))
    maxval = max(map(float, record['val'][title]))
    ymax = max(maxtrain, maxval) * 1.1
    mintrain = min(map(float, record['train'][title]))
    minval = min(map(float, record['val'][title]))
    ymin = min(mintrain, minval) * 0.9

    total_steps = len(record['train'][title])
    x_1 = list(map(int, record['train']['iter']))
    x_2 = list(map(int, record['val']['iter']))
    figure(figsize=(10, 6))
    plt.plot(x_1, record['train'][title], c='tab:red', label='train')
    plt.plot(x_2, record['val'][title], c='tab:cyan', label='val')
    plt.ylim(ymin, ymax)
    plt.xlabel('Training steps')
    plt.ylabel(ylabel)
    plt.title('Learning curve of {}'.format(title))
    plt.legend()
    plt.show()

    In [12]

plot_learning_curve(loss_record, title='loss', ylabel='CE Loss')

       

                In [13]

plot_learning_curve(acc_record, title='acc', ylabel='Accuracy')

       

                In [14]

import time
work_path = 'work/model'model = orthonet18(n_classes=10)
model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams'))
model.set_state_dict(model_state_dict)
model.eval()
aa = time.time()for batch_id, data in enumerate(val_loader):

    x_data, y_data = data
    labels = paddle.unsqueeze(y_data, axis=1)    with paddle.no_grad():
        logits = model(x_data)
bb = time.time()print("Throughout:{}".format(int(len(val_dataset)//(bb - aa))))

       

Throughout:2214

       

3. ResNet

3.1 ResNet

In [ ]

model = paddle.vision.models.resnet18(num_classes=10)
model.conv1 = nn.Conv2D(3, 64, 3, padding=1, bias_attr=False)
model.maxpool = nn.Identity()
paddle.summary(model, (1, 3, 32, 32))

   

       

3.2 训练

In [16]

learning_rate = 0.1n_epochs = 100paddle.seed(42)
np.random.seed(42)

    In [ ]

work_path = 'work/model1'model = paddle.vision.models.resnet18(num_classes=10)
model.conv1 = nn.Conv2D(3, 64, 3, padding=1, bias_attr=False)
model.maxpool = nn.Identity()
criterion = LabelSmoothingCrossEntropy()

scheduler = paddle.optimizer.lr.MultiStepDecay(learning_rate, milestones=[30, 60, 90])
optimizer = paddle.optimizer.Momentum(parameters=model.parameters(), learning_rate=scheduler, weight_decay=5e-4)

gate = 0.0threshold = 0.0best_acc = 0.0val_acc = 0.0loss_record1 = {'train': {'loss': [], 'iter': []}, 'val': {'loss': [], 'iter': []}}   # for recording lossacc_record1 = {'train': {'acc': [], 'iter': []}, 'val': {'acc': [], 'iter': []}}      # for recording accuracyloss_iter = 0acc_iter = 0for epoch in range(n_epochs):    # ---------- Training ----------
    model.train()
    train_num = 0.0
    train_loss = 0.0

    val_num = 0.0
    val_loss = 0.0
    accuracy_manager = paddle.metric.Accuracy()
    val_accuracy_manager = paddle.metric.Accuracy()    print("#===epoch: {}, lr={:.10f}===#".format(epoch, optimizer.get_lr()))    for batch_id, data in enumerate(train_loader):
        x_data, y_data = data
        labels = paddle.unsqueeze(y_data, axis=1)

        logits = model(x_data)

        loss = criterion(logits, y_data)

        acc = accuracy_manager.compute(logits, labels)
        accuracy_manager.update(acc)        if batch_id % 10 == 0:
            loss_record1['train']['loss'].append(loss.numpy())
            loss_record1['train']['iter'].append(loss_iter)
            loss_iter += 1

        loss.backward()

        optimizer.step()
        optimizer.clear_grad()
        
        train_loss += loss
        train_num += len(y_data)
    scheduler.step()

    total_train_loss = (train_loss / train_num) * batch_size
    train_acc = accuracy_manager.accumulate()
    acc_record1['train']['acc'].append(train_acc)
    acc_record1['train']['iter'].append(acc_iter)
    acc_iter += 1
    # Print the information.
    print("#===epoch: {}, train loss is: {}, train acc is: {:2.2f}%===#".format(epoch, total_train_loss.numpy(), train_acc*100))    # ---------- Validation ----------
    model.eval()    for batch_id, data in enumerate(val_loader):

        x_data, y_data = data
        labels = paddle.unsqueeze(y_data, axis=1)        with paddle.no_grad():
          logits = model(x_data)

        loss = criterion(logits, y_data)

        acc = val_accuracy_manager.compute(logits, labels)
        val_accuracy_manager.update(acc)

        val_loss += loss
        val_num += len(y_data)

    total_val_loss = (val_loss / val_num) * batch_size
    loss_record1['val']['loss'].append(total_val_loss.numpy())
    loss_record1['val']['iter'].append(loss_iter)
    val_acc = val_accuracy_manager.accumulate()
    acc_record1['val']['acc'].append(val_acc)
    acc_record1['val']['iter'].append(acc_iter)    
    print("#===epoch: {}, val loss is: {}, val acc is: {:2.2f}%===#".format(epoch, total_val_loss.numpy(), val_acc*100))    # ===================save====================
    if val_acc > best_acc:
        best_acc = val_acc
        paddle.save(model.state_dict(), os.path.join(work_path, 'best_model.pdparams'))
        paddle.save(optimizer.state_dict(), os.path.join(work_path, 'best_optimizer.pdopt'))print(best_acc)
paddle.save(model.state_dict(), os.path.join(work_path, 'final_model.pdparams'))
paddle.save(optimizer.state_dict(), os.path.join(work_path, 'final_optimizer.pdopt'))

   

       

3.3 实验结果

In [18]

plot_learning_curve(loss_record1, title='loss', ylabel='CE Loss')

       

                In [19]

plot_learning_curve(acc_record1, title='acc', ylabel='Accuracy')

       

                In [20]

##### import timework_path = 'work/model1'model = paddle.vision.models.resnet18(num_classes=10)
model.conv1 = nn.Conv2D(3, 64, 3, padding=1, bias_attr=False)
model.maxpool = nn.Identity()
model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams'))
model.set_state_dict(model_state_dict)
model.eval()
aa = time.time()for batch_id, data in enumerate(val_loader):

    x_data, y_data = data
    labels = paddle.unsqueeze(y_data, axis=1)    with paddle.no_grad():
        logits = model(x_data)
bb = time.time()print("Throughout:{}".format(int(len(val_dataset)//(bb - aa))))

       

Throughout:2232

       

4. 对比实验结果

Model	Val Acc	Parameter
OrthoNet-18	0.9406	11,270,602
ResNet-18	0.9409	11,183,562

总结

        本文发现正交核对空间压缩和获得良好的全局表示非常有用，因此本文使用正交核来替换GAP操作，取得了不错的性能。（可能CIFAR-10太简单了，性能没啥提高，后续可以换CIFAR-100等数据集试试）

# python  # git  # ai  # red  # igs  # 多个  # 最先进  # 是一个  # 取得了  # 自然选择  # 悬而未决  # 可以通过  # 将此  # 但它  # 所需要 

相关栏目： 【 Google疑问12 】 【 Facebook疑问10 】 【 网络优化91478 】 【 技术知识72672 】 【 云计算0 】 【 GEO优化84317 】 【 优选文章0 】 【 营销推广36048 】 【 网络运营41350 】 【 案例网站102563 】 【 AI智能45237 】

相关推荐： 寻宝者的发现：古董探测与文物挖掘揭秘  通义万相做小红书配图怎么用_通义万相做小红书配图使用方法详细指南【教程】  艺龙旅行AI怎样筛选最优车次_艺龙AI车次筛选与耗时最短推荐【攻略】  System of a Down：深度剖析《Hypnotize》歌词  怎么用ai进行用户画像分析 AI消费行为与兴趣标签提炼【详解】  3步教你用AI帮你把菜谱转换成详细的烹饪步骤视频脚本  v0 Report深度测评：AI文档生成器的优缺点分析与实用指南  AI在建筑行业的革命：提升效率与优化流程  ChatGPT 4.0赋能室内设计：20+实用技巧提升工作效率  Thesis AI：一键生成高质量学术论文的秘密武器  AI驱动的自动化工作流：Zapier、Perplexity和Claude集成指南  AI电影制作：颠覆传统，引领未来*新纪元  豆包Ai在线使用入口_豆包Ai官方网站最新登录地址  Power BI: 如何在 Power Query 中更改数据类型  AI如何一键生成PPT大纲_利用AI工具制作演示文稿方法【教程】  批改网AI检测工具怎样开启实时检测_批改网AI检测工具实时检测开启与延迟设置【指南】  AGI未来展望：DeepMind CEO的深度解读与行业洞察  AI图像识别如何减少保险欺诈和加速理赔  使用文心一言进行高质量的唐诗宋词创意改编  ChatGPT 如何助力建筑承包商？三大实用技巧解析  AI写作鱼怎么一键生成朋友圈文案_AI写作鱼文案风格切换与字数设置【指南】  暖心“小艺通话”：让语障人士告别沟通困境，拥抱平等生活  Kindroid AI：打造你的专属虚拟伙伴，开启AI社交新体验  教你用AI帮你写出有说服力的众筹项目文案  DeepSeek网页版怎么用_DeepSeek网页版使用方法详细指南【教程】  深度学习姿态估计：技术、应用与未来趋势全解析  掌握解方程技巧：4.2家庭作业难题精讲与分数系数处理  AI赋能！图形设计师必备的顶级AI工具  银行经理写给银行经理的信：实用模板和关键要素  超频爱好者盛宴：液氮超频Xeon 28核处理器  VHEER AI：免费在线AI图像生成器终极指南  数据迁移测试指南：策略、技术与挑战全解析  终极人声移除器UVR5：AI驱动的免费开源音频处理神器  N8N 自动化教程：HR 简历智能分析系统搭建指南  CareerCraft AI：提升大学生实习就业的智能平台  面试成功秘诀：如何巧妙回答常见面试问题  Google AI Studio Build模式更新：免费AI应用开发新纪元  MediCa AI：AI赋能的智能医疗保健平台全面解析  N8N工作流：自动化知识管理与智能问答解决方案  如何通过 DeepSeek 优化 Kubernetes 配置文件  百度AI搜索怎样设置搜索偏好_百度AI搜索偏好设置与个性化推荐【技巧】  Character AI深度解析：功能、用户反馈与替代方案全攻略  通义万相IP形象设计怎么用_通义万相IP形象设计使用方法详细指南【教程】  提升效率的AI工具：Jace、Yutori、Dia等效率神器测评  文心一言怎么一键生成会议纪要_文心一言纪要生成与重点提取【指南】  在线歌曲歌词生成器：创意歌词轻松创作指南  即梦AI怎样生成产品描述_即梦AI产品描述生成与卖点提炼【实操】  百度ai助手快捷键怎么关 百度ai助手快捷键取消设置  AI内容检测与优化：免费工具助你提升内容质量  生物医学图像分割：U-Net模型训练与应用详解 

 2025-07-31