Imagenet1k数据集导入

发表于 更新于

下载imagenet1k数据集

链接: https://pan.baidu.com/s/1L5CSPffnVmeepC6pK35vbQ?pwd=abel 提取码: abel

数据格式配置

创建train,val,test文件夹,把压缩包放入各文件夹中,解压各个压缩包文件.其中train压缩包会产生1000个子压缩包,删除原压缩包,输入以下shell命令进行批量解压缩 

1
2
3
4
5
6
7
8
for f in *.tar; do [ -f "$f" ] && mkdir -p "${f%.tar}" && tar -xf "$f" -C "${f%.tar}"; done && rm *.rar
```    
执行脚本之后，我们就获得了train文件夹下1000个文件夹和对应的图片数据
## 数据标记映射
1. 先解压devkit压缩包
    
    ```python
    tar -xzf ILSVRC2012_devkit_t12.tar.gz

  1. 之后，在imagenet目录（devkit和val的根目录下）创建并运行如下 python 脚本

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    from scipy import io
    import os
    import shutil

    def move_valimg(val_dir='./val', devkit_dir='./ILSVRC2012_devkit_t12'):
        """
        move valimg to correspongding folders.
        val_id(start from 1) -> ILSVRC_ID(start from 1) -> WIND
        organize like:
        /val
        /n01440764
            images
        /n01443537
            images
            .....
        """
        # load synset, val ground truth and val images list
        synset = io.loadmat(os.path.join(devkit_dir, 'data', 'meta.mat'))
        
        ground_truth = open(os.path.join(devkit_dir, 'data', 'ILSVRC2012_validation_ground_truth.txt'))
        lines = ground_truth.readlines()
        labels = [int(line[:-1]) for line in lines]
        
        root, _, filenames = next(os.walk(val_dir))
        for filename in filenames:
            # val image name -> ILSVRC ID -> WIND
            val_id = int(filename.split('.')[0].split('_')[-1])
            ILSVRC_ID = labels[val_id-1]
            WIND = synset['synsets'][ILSVRC_ID-1][0][1][0]
            print("val_id:%d, ILSVRC_ID:%d, WIND:%s" % (val_id, ILSVRC_ID, WIND))

            # move val images
            output_dir = os.path.join(root, WIND)
            if os.path.isdir(output_dir):
                pass
            else:
                os.mkdir(output_dir)
            shutil.move(os.path.join(root, filename), os.path.join(output_dir, filename))

    if __name__ == '__main__':
        move_valimg()

  2. 数据加载
    使用 torchvision.datasets.ImageFolder() 就可以直接加载处理好的数据集啦！

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    import os
    import torch
    import torchvision.datasets as datasets

    root = 'data/imagenet'
    def get_imagenet(root, train = True, transform = None, target_transform = None):
        if train:
            root = os.path.join(root, 'train')
        else:
            root = os.path.join(root, 'val')
        return datasets.ImageFolder(root = root,
                                transform = transform,
                                target_transform = target_transform)