[ale] Using a namespace to manage a chroot.

[Jim Kinney]

Yeah. Big changes from i586 to i686. I had forgotten your, um, embedded in the embedded space ;-). You and I work at opposite ends of the Linux world! Totally different challenges.

Code bloat. Install kernels are huge. But to keep security patches respected, custom kernels are not used. The loss of ram is not as costly as the support time. But for super small systems ram size is a critical issue so every byte removed is a win.

Lib sizes have increased as the cpu instruction set size has increased. 

Your build process looks very exacting. I've not faced the issues you have with emulating much older cpu and having the real newer one's specs bleed through. 

You should write a book on working in tiny compute space with obsolete hardware. What you have constructed is quite unique and the principles are  applicable across several regions. Of course only in your copious spare time :-)

On May 9, 2021 11:25:15 PM EDT, Chris Fowler <cfowler at outpostsentinel.com> wrote:
>Correct,  hardware not supported by a modern distro.   i586 that looks
>like this:
>
>processor : 0
>vendor_id : Genuine  RDC
>cpu family : 5
>model : 8
>model name : 05/08
>stepping : 6
>cpu MHz : 999.924
>fdiv_bug : no
>hlt_bug : no
>f00f_bug : no
>coma_bug : no
>fpu : yes
>fpu_exception : yes
>cpuid level : 1
>wp : yes
>flags : fpu tsc cx8 mmx up
>bogomips : 1999.84
>clflush size : 32
>cache_alignment : 32
>address sizes : 32 bits physical, 32 bits virtual
>power management:
>
>Not much variety in the instruction list.  The newest Ubuntu disti that
>will run on it is !0.04.  I did not notice how old CentOS 5 is until I
>built newer disti that was still i586.  Thats the LFS-7.7 in the script
>and I'd it runs newer packages than CentOS 7.
>
>The 2nd CentOS 5 root is a pristine copy of the same install as the
>first, but only used for svn co before it is backed up.  The first C%
>root is the work area and dirty.
>
>The challenge is maintaining the i586 build as i586 on a system that is
>really x86_64.  /bin/uname is not the only way to identify a system. 
>The C5 roots and even the LFS-7.7 root has a fake /bin/uname that will
>determine if I'm building.  If so, it returns BS info to the caller. 
>If not, it execs the real one from coreutils.  Next, I run a module
>named 'unam_hack'.  It replaces utsname() in the running kernel.  This
>works great, but do not do an apt-get update in the host.  The whole
>desktop reports back as i586.
>
>Those two methods have worked great.   I believe that GCC compiles some
>assembly ad-hoc and runs it to query the CPU.  I can't fake that.  I
>can inform GCC during a normal compile to pass arguments to AS that
>will stop the build if any contra-band instruction tries to get past. 
>If a C program uses inline assembly to execute CMOV, AS will error and
>refuse assembly.   I can then look at the offender and try to determine
>why it is doing that even though it's on configure told it to build on
>i586.  Everyone will tell you to just leave targets, CFLAGS, LDFLAGS,
>etc to ./configure, but I'll tell you that the Makefile of many
>packages have problems with authority, so I like to "head them off at
>the pass" but forcing specific behavior in the development tools.  If
>you build 700 packages and prefix each one with CFLAGS I guarantee that
>they will be offenders who will ignore you and tell you to pound sand
>on that idea.  This is especially true if the libraries you are
>building against are somewhere else than /lib and /usr/lib.  The only
>thing not built from scratch is /lib/libc and a few friends.  I am
>still taking them from the dev host.
>
>To make the LFS-7.7 as true to i586 as possible and not wait a week, I
>built the /tools (what is used to build the disti) using the native
>hardware and told GCC to go 'native'.  That build takes 23 hours.  The
>next stage (build the whole distribution) on the native hardware takes
>63 hours.   My workstation will build it in 2.5 hours using Samsung
>fast SSD, and all 12 threads going to make -j12.  Did I fool it enough
>that there are no illegal instructions?   I'll do a search for all
>ELFs, dissamble them, and the search that for illegal instructions. If
>I find one, I have to go to the source and fund out why.
>
>That hardware runs OpenSSL 1.1.1f with all the CVEs applied.  I had to
>patch many of the other programs because they were written for 0.9.8f
>and 1.1.1f  had some major changes in architecture.  Some pks I was
>able to update, and they would still compile with GCC 4.1.2.  Seme PKGs
>require the SCL toolset #2 to compile.
>
>The downside to the newer software is that it brings in bloat.  The
>image that runs on the device is only 50M in size and runs as a
>ramdisk.  Device has 1G of ram. The tempalte (product of a full build)
>I create that 50M squashfs image off of is 2.2G in size.  I use a
>packaging system during build which created dev, doc, extra, etc
>packages for each package compiled.   To make the image, I rsync copy
>the template.  I then back out all the packages that will not be
>needed.  That includes all those marked as 'dev'.   I need headers and
>other stuff to do the build.  50M goes to "firmware".  The remaining
>are PKG tarballs that are uploaded to a repo.   I can use a program on
>the device that can download off the repo and store on ext4 with
>persistent data, like logs.   At each boot, the pkgs in that directory
>are installed into /, a union of /dev/ram0 (ro) and a tmpfs (rw).  The
>goal here is to never worry about data integrity due to power loss. 
>Simply pull the plug!  Rest the box and it is as it was when it last
>booted.  Pristine.  The logs space is not mounted as a union into /,
>like what Ubuntu does with the casper image on a USB stick.   That
>means that most of /etc is built each boot out of a config system.  If
>a device fails, you simply replace it with another and then 'restore
>cloud' using the failed device's serial.   I got sick of bricking units
>during development with my mistakes, so I added that little feature.
>
>The i586 devices are not made by the manufacturer now.   I still have a
>few hundred deployed so I' trying to support them a little longer.  I
>use a J1900 device a a replacement that runs the same firmware image. 
>My plan is to switch from C5 to the LFS-7.7 root for building the
>firmware.   Test and cut a release from within that root.  Test the
>i586 device under the LFS-8.0 i586 root.  I will split off the J1900
>and go x86_64 on it.  The RDC will only get minor updates, no features.
>
>What's neat about that LFS-8.0 is that I've scripted it just like I did
>the LFS-7.7.  To have LFS-8.0 i586, you need LFS-7.7 i586.  To have
>that, you need Ubuntu 10.04-4 i386. It's all scripted and chained to
>create an i586 and x86_64 distis for each.
>
>On my Ubuntu 18.04 x86_64 I'll insmod uname_hack.ko and then run
>make-disit-all.
>
>  1.   Downloads a tarball of a / of Ubuntu 10.04-4
>  2.   Unpacks into a directory.
>  3.   Downloads all the build scripts and sources for LFS 7.7
>  4.   chroot into that and builds /tools and then the whole disiti.
>5.  if KEEP_GOING=0 it starts sshd using daemon in the LFS 7.7 root and
>then stops.  I can now ssh into a pristine root to validate.
>6.  KEEP_GOING=1 means it downloads the scripts needed to make 8.0 and
>the sources
>  7.  It builds /tools using its environment
>8.  It chroots into a directory that only contains /sources and the
>/tools and it uses /tools it just built to build 8.0
>
>You could do it in large steps all the way up to LFS 2021 as long as
>GCC 10 does not tell you to pound sand using i586.
>
>
>You could use a project like buildout to do some of this, but the RDC
>is not an anemic device.    The original intention of the manufacture
>is to use the RDC as thin clients, POS, etc.  They'll run Windows for
>that CPU and they'll run any Linux disti that supports i586 and
>non-PAE.  When I talk about the RDC I call it an "embedded" device.  
>It is treated like an appliance, but more resources than most
>appliances built during its time.
>
>I still have java related project that is till 32 bit and runs in
>CentOS 5 or 6.  I'm in the process of upgrading it to 64bit.  It's a
>big project because it contains everything required to run the web app
>while only needed access to /lib on the host for a few items.  I guess
>it was like snaps, before snaps was a thing.  I can't really get a
>CentOS 6 droplet from Digital Ocean now.  I can install one in VMWare,
>upload the vmdk, but they don't real support it.  They also require
>cloud-init-0.7.7, C6 runs 0.7.5.    I have a few C6 droplets that have
>been running for 7 years at DO.  I upgraded the oldest (6.5) to 6.10
>and created a snapshot.  DO allows you to build a droplet off a
>snapshot.  That 6.5 was so old that it was using dracut.    The system
>accepted the snapshot for the new droplet, but there was no data center
>that could run it.  I uploaded a pristine C6 vmdk image and ran into
>issues booting it to net.  Using the web console, I was able to fix all
>that.   I don't feel good about it and DO does a great job of making it
>clear that you will always be solely responsible for custom images.  I
>can only take C6 so far.  I have LFS-7.7 close to being ready for a
>droplet.  Cloud-int, etc.  DO has great documentation, but every email
>is more about "we don't support custome images and we are doing you a
>favor by sending a reply".
>
>It's a non-issue really because I'm far along in the x86_64 upgrade for
>that project.  When it comes down to brass tacks with "if it ain't
>broke you don't fix", as long as you can address CVEs , you don't do
>something so drastic as a tear down and replace.  You address the CVE. 
>Moving from OpenSSL 0.9.8f on the i586 device to 1.1.1f allows me to
>address the ones that a truly exploitable on the device.  If you're in
>the business of distributing rootkits, you need to target the lowest
>common denominator and be prepared that a reboot makes it as if you
>were never there to even get onto that RDC.  The life of your rootkit
>is limited to that boot.
>
>
>
>
>
>________________________________
>From: Jim Kinney <jim.kinney at gmail.com>
>Sent: Sunday, May 9, 2021 7:05 PM
>To: Atlanta Linux Enthusiasts <ale at ale.org>; Chris Fowler via Ale
><ale at ale.org>; ale at ale.org <ale at ale.org>
>Cc: Chris Fowler <cfowler at outpostsentinel.com>
>Subject: Re: [ale] Using a namespace to manage a chroot.
>
>Wow. Really cool stuff. Jchroot looks like a way to solve some real
>bozo uses of containers and perhaps a non-cgroup way of isolating
>processes for HPC work.
>
>Why still centos 5? Specific hardware issues with later stuff?
>
>On May 9, 2021 4:47:50 PM EDT, Chris Fowler via Ale <ale at ale.org>
>wrote:
>For years I've been running SSH via chroot inside Linux installs on my
>workstation regardless of the version of Ubuntu the workstation
>currently runs.  This allows me to upgrade my workstation, while still
>compiling code inside a CentOS distribution.
>
>At boot I'll do something like this to prepare each chroot.
>
>START_PORT=55;
>for ii in CentOS-5-1 CentOS5-2 LFS-7.7; do
>  for iii in dev dev/pts proc sys; do
>    TEMPLATE="/opt/devel/${ii}"
>     mount -o bind /${iii} ${TEMPLATE}/${ii}
>sed -i 's#^Port 22.*$#Port '${START_PORT}'#g'
>${TEMPLATE}/etc/ssh/sshd_config
>     chroot ${TEMPLATE}/${ii} /etc/init.d/sshd start
>     START_PORT=$(( ${START_PORT} + 1 ))
>  done
>done
>
>After boot,  My Ubuntu 18.04 workstation will be running 3 other
>distributions.  I'll use SSH to access them as a regular user.  Tmux
>automatically runs on first login, other logins will attach to that
>session.
>
>I am thinking I can use unshare to create a namespace and a control
>group attached to the SSHD start.   On death of the SSHD, all processes
>would automatically be killed, and all mounts be unmount.  I'm not 100%
>sure how to change over to this.    Docker is a no-go because
>containers are not designed to run a whole disti that requires
>persistent storage on its /.   My workstation supports KVM and I
>considered this, but when I compile in these system I need all the
>power the workstation can give in CPU and I/O.
>
>If I need to run 'ssh -D' in each one instead for the unshare, I could
>use daemon to do it.  On the host, I could just use 'daemon
>--name=sshd-1 --stop' to tear down the chroot?
>
>I found this last night https://github.com/vincentbernat/jchroot
>
>It is interesting and works fine under a running 2.6.38 kernel. 
>Creates mounts, runs /bin/bash, on exit the mounts do not exist.   On
>4.15.0 it does not tear down the mounts on exit.   I don't see any
>errors during strace either.   I could use that as the program daemon
>runs to start the SSD.  daemon  --name=devel-sshd01  -- /sbin/jchroot
>-f /opt/devel/CentOS-5-1/etc/fstab  /opt/devel/CentOS-5-1
>/usr/sbin/sshd -D -p 55
>
>To tear it all up: daemon --name=devel-sshd01 --stop
>
>
>
>[https://avatars1.githubusercontent.com/u/631446?s=400&v=4]<https://github.com/vincentbernat/jchroot>
>GitHub - vincentbernat/jchroot: a chroot with more
>isolation<https://github.com/vincentbernat/jchroot>
>jchroot: a chroot with more isolation. Recent Linux kernels are now
>able to provide a new PID namespace to a newly created process. The
>process becomes PID 1 in its own namespace and all processes created in
>this namespace will be killed when the first process terminates.
>github.com
>
>
>
>
>--
>Computers amplify human error
>Super computers are really cool

-- 
Computers amplify human error
Super computers are really cool
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.ale.org/pipermail/ale/attachments/20210510/2dfcc8e4/attachment.htm>