finding plain text files vs. binary files on unix or linux - this is surprisingly tough.

Показать описание

Yo what's up everyone my name's dave and you suck at programming.

🔗 More Links

📖 Keywords
you suck at programming #programming
#devops #bash #linux #unix #software #terminal #shellscripting #tech #stem

Рекомендации по теме

Комментарии

The real answer is "send it back to the business analyst and ask for better defined acceptance criteria"

daddysolaire

You just turn a plain text file into a ship of theseus.

daddysolaire

It's not an easy or hard problem; it isn't well defined in the first place.

All files are binary - they consist of bytes; that's the only thing that can be held in memory or stored on disk. Text is *one way* of interpreting that binary content, and not even a well-defined way in and of itself; you need an encoding. You can check whether some data is valid according to the encoding rules, but lots of text encoding rules say every byte is always meaningful. You need more heuristics after that to decide whether the resulting interpreted text actually makes sense to use as text. And then obvious rules ("are these lines super long?) might not be relevant depending on what you want to use the text for.

la.zanmal.

Not to be a complainer, but if it were me and I were making a video (which I’m not) I would at least research what an executable file header looks like prior to taking to the Internet to pontificate. On the other hand, thanks for all the free knowledge!

gregoryshields

`find -type f -exec file {} \; | grep "text"` will help you filter down to the files that are more likely text, though.

You're right, it's a nearly impossible question, but `file` does 95% of the work for you.

greyfade

I love just testing things out and seeing where i go. Then never having to do that ever again and completely forgetting how i did it before… good times… good times

jeh_io

I'd start with the file command (it checks the magic number) But clearly I've not plumbed the depths of this problem or considered bad actors.

robertkielty

You might be able to get around using the file command. That determines the mine type if you're lucky... But it can also be tricked...

gerke_kok

Most western languages have a thing for vocals. And most files are written in western languages. (Not only prose, also source code you have to consider)
Now do some statistics summing up the amounts of every value of every byte. Does is have the ascii code for e and an and i in it unusually often? Then consider this a text file.

Or just use grep.

hansdampf

if i remember correct there was three magic byte 🤔🧐

Kalvi-Karing

In my mind: printer driver not working on some particular day beause of bug in file command

kf

The magic number is arbitrarily long, and some file formats even I actually have more than one magic number, HTML comes to mind. [Html has either <!DOCTYPE html> or <html> as its magic numbers] (in fact, the whole point of the doctype tag is to tell the HTML interpreter. What type of document it has, because XML and SVG are things. Why is XML a thing still?
The issue is a character is a character because ASCII takes up the whole bite, so a validly formatted ANSI stream buff is indistinguishable from any other binary data. [That being said, a. "Plain text" file probably won't have any code points less than 0x20 But it would be valid ASCII for them to.

skeleton_craftGaming

is for sure that in UNIX system everything is a file.

ArmandoCalderon

Why not to use file utility, it tells what the file is (image, text, ....)

amorsmor

Doesn’t the command file have this built in?

SupportToolsBlog

how about: index all files and create a database table or a json file with the following schema: filename, directory_path, plaintext_percent

jaytea

of they arwnt compressed surely entropy is a good.measure and could probably reliably tell you if something is text or binary

you could definitelt.also.ise some machune learning library tools for this

CosumuHcos

Man, you're outside with nice weather, most likely some grass around to touch. And you're talking about this nonsense.

mikemiller

finding plain text files vs. binary files on unix or linux - this is surprisingly tough.

finding plain text files vs. binary files on unix or linux - this is surprisingly tough.

0302: Advantages/disadvantages of plain-text files

How To Use Compare & Find Differences Between Two Text Files Software

Plain Text Is Awesome

6. Finding files, directories and text super quickly | Linux Essentials

How to Compare Text Online to Find the Difference between Two Text Files

Use Diff Tools to Compare Text Files and Spot the Differences (Softpedia App Rundown #61)

Find a file that contains certain text - Linux Mint / Ubuntu

week5_summary_T2_may2025

6 Weltner Sophisitcated Techniques of Plain Text Parsing

Finding same and different data from text file in C#

Comparing Two Text Files in PHP: A Helpful Guide to Finding Similarities

How to Use VS Code Extensions to Find Text in Files Efficiently

Importing and Munging Plain Text Data in Python

How to Fetch Salary by Name Using VLOOKUP in Excel – Easy Guide 💼 #excel #vlookup

How to see hidden files /folder (part :- 2) #shorts #youtubeshorts

This will Clean Virus from your PC 😱😱 #shorts #windows

Section 2.8: “Finding and replacing” from Learn Enough Text Editor to Be Dangerous

How to find all files containing specific text (string) on Linux - Linux Tips And Tricks

DON'T buy a Kindle before watching this😳#tech

The WORST Message on ROBLOX!😳😭#shorts

Top 3 Password Cracking Tools 🛠️

How To Escape AI Detector Effortlessly?

How to use VLOOKUP in Excel! #excel