[ale] Piping, redirection and shellscipts: 3/5/2025 7pm Eastern Standard time

[DJPfulio at jdpfu.com]

When ever bash becomes over 1 pages long, I switch to a real scripting language that has data structures that aren't torture to use.

I routinely snag webpage source and use egrep to get just the URLs from it.  That's a single command, thanks to the -o egrep option.  I wouldn't keep anything in YAML if I needed to massively update it later.  Creating YAML from formatted text/csv is really easy.

As for the source of the data being in a webpage, we don't have control over past workers and lots of people will happily point-n-click to enter data for 5 days rather than create a 10 line script to do it in 10 seconds.

Came across some Solaris Admins like that who insisted adding 6000 printers to about 40 servers would take them over a year, based on their Solaris training.  I looked up the config file format used by printers and wrote about 10 lines of perl to generate that file for the 6000 printers in less than 1 hour (think it was really 10 minutes, since they already had the printer names/models/IPs in a text file). Had them copy the output config file into each server and we were done.  Not everyone thinks that way.  After all, Sun Microsystems taught them to point-n-click to add a printer, so there's no other, shorter, faster, method that could possibly work.

To be fair, sometimes we all do things the hard way.  My XMLTV building script is probably overly complex, but since it mostly works and I like the grid that data creates for Jellyfin so it can be used like a TiVo recording device without paying SD $35/yr is good.

On 3/3/25 05:42, Ron via Ale wrote:
> Steve Litt via Ale wrote on 2025-03-02 15:00:
> 
>> Yesterday I was tasked with getting every domain name I owned into
>> my list at 444domains.com/domains , which is created by a
>> shellscript and Python program that read a Yaml file and convert it
>> to the web page. So I had to do the following:
>>
>> * Screen scrape all my domain names from ionos, put in file
>>
>> * Screen scrape all my domain names from Spaceship.Com, put in
>> another file
>>
>> * Use Vim to remove everything but the domains themselves
>>
>> * Sort both files
>>
>> * Create a Python program to sort my Yaml file
>>
>> * Run diff between each file and the sorted domains of the Yaml file - The output with the left arrows need to be added to Yaml file
>>
>> * Re-sort the Yaml file
>>
>> * Run the shellscript to create the web page.
>>
>> This sounds straightforward
> 
> No, it does *not* sound straightforward!
> 
> 
> Is it just me, or does it sound like a Rube Goldberg contraption in
> software?
> 
> 
> 
> The list of all domain info should be in a DB or at least a text file,
> for starters.
> 
> 
>  From there, do the sort(s), build the YAML, etc. Why sort a YAML file?!?
> Running `diff` and manually merging?
> 
> 
> All this scraping seems pointless, why is the source of this data inside
> web pages?
>