前段时间需要验证某客户的 32 位 ARM Linux 程序，由于我们只有现成的 ARM64 设备和系统，而 ARM64 兼容 32 位 armel 和 armhf，因此想到直接在 ARM64（aarch64）Linux 系统中运行 32 位 ARM（armhf）容器来进行测试。

客户程序使用的是 CentOS 7 32 位 ARM 开发环境，这里我使用 LXC 官方镜像来创建一个完整的 CentOS 7 容器系统：

~ # lxc-create -n arm -t download --dir=/var/lib/lxc/arm/rootfs -- --server mirrors.tuna.tsinghua.edu.cn/lxc-images -d centos -r 7 -a armhf

32 位 ARM 容器可以正常启动，但使用 yum 安装任何软件包会报错：

bash-4.2# yum install openssh-server
Loaded plugins: fastestmirror
Determining fastest mirrors


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/7/armv8l

yum 修改

上面 yum 命令的报错提示找不到 repo 为 base/7/armv8l 的地址，我们可以在容器中使用 uname 命令确认当前的系统架构为 armv8l（ARM64 内核运行 32 位 ARM 程序）：

bash-4.2# uname -m
armv8l

而在主机端则可以看到真正的系统架构为 aarch64：

~ # uname -m
aarch64

正是由于 yum 不能正确识别 armv8l这个特殊的系统架构，导致命令执行失败。

我们可以修改 /usr/lib/python2.7/site-packages/rpmUtils/arch.py 文件，使 yum 可以正确处理 armv8l 架构，以下为 patch 修改内容：

diff -rNp a/arch.py b/arch.py
*** a/arch.py   Thu Oct 29 23:46:21 2020
--- b/arch.py   Thu Oct 29 23:48:55 2020
*************** arches = {
*** 73,78 ****
--- 73,79 ----
      "armv5tel": "noarch",
  
      #arm hardware floating point
+     "armv8l": "armv7hl",
      "armv7hnl": "armv7hl",
      "armv7hl": "noarch",
  
*************** def getBaseArch(myarch=None):
*** 442,447 ****
--- 443,450 ----
          return "ppc"
      elif myarch.startswith("arm64"):
          return "arm64"
+     elif myarch == "armv8l":
+         return "armhfp"
      elif myarch.startswith("armv7h"):
          return "armhfp"
      elif myarch.startswith("arm"):

此时再运行 yum 就可以正确找到软件包了：

bash-4.2# yum install openssh-server
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
base                                                                                                                                                                                              | 3.6 kB  00:00:00
extras                                                                                                                                                                                            | 2.9 kB  00:00:00
updates                                                                                                                                                                                           | 2.9 kB  00:00:00
(1/4): base/7/armhfp/group_gz                                                                                                                                                                     | 153 kB  00:00:00
(2/4): extras/7/armhfp/primary_db                                                                                                                                                                 | 170 kB  00:00:00
(3/4): updates/7/armhfp/primary_db                                                                                                                                                                | 562 kB  00:00:00
(4/4): base/7/armhfp/primary_db                                                                                                                                                                   | 4.1 MB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package openssh-server.armv7hl 0:7.4p1-21.el7 will be installed
--> Processing Dependency: libwrap.so.0 for package: openssh-server-7.4p1-21.el7.armv7hl
--> Running transaction check
---> Package tcp_wrappers-libs.armv7hl 0:7.6-77.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================================================================================================
 Package                                                    Arch                                             Version                                                 Repository                                     Size
=========================================================================================================================================================================================================================
Installing:
 openssh-server                                             armv7hl                                          7.4p1-21.el7                                            base                                          446 k
Installing for dependencies:
 tcp_wrappers-libs                                          armv7hl                                          7.6-77.el7                                              base                                           64 k

Transaction Summary
=========================================================================================================================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 510 k
Installed size: 1.0 M
Is this ok [y/d/N]: y
Downloading packages:
(1/2): tcp_wrappers-libs-7.6-77.el7.armv7hl.rpm                                                                                                                                                   |  64 kB  00:00:00
(2/2): openssh-server-7.4p1-21.el7.armv7hl.rpm                                                                                                                                                    | 446 kB  00:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                    1.7 MB/s | 510 kB  00:00:00
Running transaction check
Running transaction test


Transaction check error:
  package tcp_wrappers-libs-7.6-77.el7.armv7hl is intended for a different architecture
  package openssh-server-7.4p1-21.el7.armv7hl is intended for a different architecture

不过这次轮到 rpm 报错了，提示 RPM 包的架构不相符。

rpm 修改

我们先试着手工安装 yum 下载下来但安装失败的 RPM 包，可以看到还是相同的报错：

bash-4.2# rpm -ivh /var/cache/yum/armhfp/7/base/packages/tcp_wrappers-libs-7.6-77.el7.armv7hl.rpm
Preparing...                          ################################# [100%]
        package tcp_wrappers-libs-7.6-77.el7.armv7hl is intended for a different architecture

这也是由于 rpm 命令不能正确处理 armv8l 架构，其实对于这种 RPM 包的架构不相符的情况，我们可以为 rpm 命令增加 --ignorearch 参数强制安装：

bash-4.2# rpm -ivh --ignorearch /var/cache/yum/armhfp/7/base/packages/tcp_wrappers-libs-7.6-77.el7.armv7hl.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:tcp_wrappers-libs-7.6-77.el7     ################################# [100%]

当然我们的最终目的是要让 yum 能够自动安装软件包，因此还需要让 rpm 命令也能正确识别 armv8l 架构。

为此我们需要修改 /usr/lib/rpm/rpmrc 文件，将 armv8l 视为 armv7hnl：

diff -rNp a/rpmrc b/rpmrc 
*** a/rpmrc     Fri Oct 30 00:15:17 2020
--- b/rpmrc     Fri Oct 30 00:16:32 2020
*************** arch_compat: armv5tel: armv4tl
*** 398,403 ****
--- 398,404 ----
  arch_compat: armv4tl: armv4l
  arch_compat: armv4l: armv3l
  arch_compat: armv3l: noarch
+ arch_compat: armv8l: armv7hnl
  arch_compat: armv7hnl: armv7hl
  arch_compat: armv7hl: noarch

此时我们再使用 yum 命令就可以正确下载和安装软件包了：

bash-4.2# yum install openssh-server
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package openssh-server.armv7hl 0:7.4p1-21.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=========================================================================================================================================================================================================================
 Package                                                  Arch                                              Version                                                 Repository                                      Size
=========================================================================================================================================================================================================================
Installing:
 openssh-server                                           armv7hl                                           7.4p1-21.el7                                            base                                           446 k

Transaction Summary
=========================================================================================================================================================================================================================
Install  1 Package

Total size: 446 k
Installed size: 921 k
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : openssh-server-7.4p1-21.el7.armv7hl                                                                                                                                                                   1/1
  Verifying  : openssh-server-7.4p1-21.el7.armv7hl                                                                                                                                                                   1/1

Installed:
  openssh-server.armv7hl 0:7.4p1-21.el7

Complete!

后续安装其它软件包也都没有问题了，例如安装基础开发包：

bash-4.2# yum -y install gcc make openssl-devel glibc-static openssl-static

Error: Apache has not been designed to serve pages while running as root. There are known race conditions that will allow any local user to read any file on the system. If you still desire to serve pages as root then add -DBIG_SECURITY_HOLE to the EXTRA_CFLAGS line in your src/Configuration file and rebuild the server. It is strongly suggested that you instead modify the User directive in your httpd.conf file to list a non-root user.

看来 Linux 发行版中的 apache 因为安全原因默认就不允许以 root 身份运行了，必须修改编译选项 BIG_SECURITY_HOLE 重新编译安装 apache 才可以。

由于我使用的是 64 位的 RedHat Enterprise Linux 6 系统，考虑直接使用源代码 RPM 包的形式编译，我们实际上只需要替换 apache 的 httpd 这个二进制文件就可以启用 root 权限了，这样是最简单方便的方法了。

首先确认当前 apache 的版本：

rpm -q httpd
httpd-2.2.15-9.el6.x86_64

下载同样版本的 source rpm 包：

cd /usr/src/redhat/SRPMS
wget http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/httpd-2.2.15-9.el6.src.rpm

然后使用 rpm2cpio 命令解压缩 RPM 包：

rpm2cpio httpd-2.2.15-9.el6.src.rpm | cpio -iduv

接着从解压缩出来的源代码中可以看到源代码包：httpd-2.2.15-9.tar.bz2 还有一堆 patch 补丁文件，另外就是比较重要的编译配置文件 httpd.spec 了。

这时就可以修改 httpd.spec 文件中的编译选项了，根据最开始的提示找到 CFLAGS，并加上 BIG_SECURITY_HOLE 定义：

CFLAGS="$RPM_OPT_FLAGS -Wformat-security -fno-strict-aliasing -DBIG_SECURITY_HOLE"

保存退出，接着就可以直接使用 rpmbuild 命令直接编译出 RPM 包了：

rpmbuild --ba httpd.spec

稍微一会就可以找到编译出的 RPM 包了：

/root/rpmbuild/RPMS/x86_64/httpd-2.2.15-9.el6.x86_64.rpm

从编译出的 RPM 包里解压缩出 httpd、httpd.event、httpd.worker 这三个文件替换到 /usr/sbin 等目录下，再重启 httpd 服务就可以使用 root 身份运行 apache 了。