Qvm-convert-pdf crashes exactly on the 50th page

shimmy_jimmy · September 7, 2024, 2:37pm

Hi. I noticed that qvm-convert-pdf crashes exactly on the 50th page no matter what pdf file it is but with increased image quality in template. I tested 4 different pdf files which had above 50 pages (55-215) and that tool is crashing with Killed message exactly on the 50th page. With pdf files that have less than 50 pages, it is converting correctly.
What I changed before using qvm-convert-pdf:

created different fedora template for pdfs.
created different disposable template for pdfs.
increased image quality by adding "-r", "500", in 148 line (create_irep()) of server.py here:/usr/lib/python3.12/site-packages/qubespdfconverter in fedora template for pdfs.
changed default disposable template in a qube in which I have that pdf files to that created for pdfs which is important because qvm-convert-pdf runs disposable vm to convert pdf.
Also what is important - it is crashing when I set dpi to >= 500. It’s not crashing with Killed message when I set it to 300-400 for example. Also I monitored RAM and CPU usage and it was fine so it is not caused by some out of memory-like errors. Strange thing is that it is crashing exactly on 50th page.

apparatus · September 7, 2024, 2:46pm

This is related issue:

github.com/QubesOS/qubes-issues

Convert to Safe PDF crashes when "fancy" PDFs are tried

opened 06:53PM - 08 Jun 21 UTC

closed 02:36AM - 08 Jul 24 UTC

ninavizz

T: bug C: other ux P: default diagnosed pr submitted affects-4.1

**The problem you're addressing (if any)** In a recent user research interview,… a user relayed to me that PDFs they had been sent by unknown 3rd parties (so, precisely what this tool was built to process) do not work with this tool and effectively crash it, when those PDFs are "too fancy." When I inquired about what that meant, exactly, the user cited the plethora of advanced PDF features Adobe is able to embed in PDFs when those PDFs are created with its own tooling. Other apps may also leverage those advanced capabilities built and licensed by Adobe. I do not know. I do recall as a former Adobe employee, that they have major contracts with State defense departments and multiple other B2B customers to help them embed interactivity, signature collection, and possibly embedded videos—and probably interactive & tracking stuff I never noticed before—in PDFs generated by their tools. **Describe the solution you'd like** Talented developers looking at the code and fixing it up such that the most bloated and ridiculous PDF fails to crash this tool. Regrettably, my own talents fall short of that. **Where is the value to a user, and who might that user be?** All users: sparing folks the "FAFO" trial-and-error disappointment of a tool not delivering on its implied promise. Non-Linux-Native users, and newbies to Libre stuff: a better experience for users unaccustomed to reading documentation before diving into use a tool/utility. **Describe alternatives you've considered** Currently this user observes that `Convert To Trusted PDF` gets hung-up or crashes. His workaround, is to then "Print to PDF" from a viewer app in a dvm, and then he runs _**that**_ output through `Convert To Trusted PDF`. At a minimum, it'd be nice if the utility could detect such problematic attributes in PDFs in a pre-flight examination of the document, and then message users with this proposed workaround before attempting to process the PDF. Happy to work with anyone interested in taking this on, to suggest user-facing things. **Additional context** For Reasons I cannot publicly share PDFs that caused this, regrettably—but I have asked the user to keep in touch with me as they are able to, so that I can learn more. I will share those insights here, should they happen. **Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues** #6181 also proposes an unrelated enhancement to Trusted PDF converter

It was fixed, but I guess it was broken again at some point after update?
Can you report this in this github issue so it’ll be reopened?

alimirjamali · September 7, 2024, 3:05pm

Would you please check the output PDF size. Is it over 2GB or nearly 2GB.

p.s.: Also what is the page size of the input PDF.

alimirjamali · September 7, 2024, 3:27pm

BTW, This is the PR I submitted in July to make the output PDF resolution configurable. It awaits review:

shimmy_jimmy · September 7, 2024, 6:27pm

I cannot check the output PDF size since there are no output PDFs. It is being killed during sanitizing process so it is not completed and I couldn’t find any directory where pdftocairo works in disp VM (in order to find some incompleted PDF file). Input PDFs page size: 595 x 842 - A4 or 612 x 792 - letter, but I don’t think that matters as I checked it on few different PDFs with different page sizes and if PDF has > 50 pages, it is being Killed at 50th page.

shimmy_jimmy · September 7, 2024, 6:28pm

Thanks for response, I will try to report it.

tempmail · September 9, 2024, 4:42pm

I’ve noticed yesterday updating dom0, there is a new version

qubes-pdf-converter-dom0.noarch 2.1.21-1.fc37

Did you try it?

apparatus · September 11, 2024, 4:45am

Can you post the link or upload the PDF that is causing this issue for you?
I’ve tried converting the large PDF that was linked as an example in the github issue and it worked for me.

shimmy_jimmy · October 21, 2024, 8:03am

Did you convert it with increased image quality in /usr/lib/python3.12/site-packages/qubespdfconverter/server.py? If I convert it without increased image quality (150 dpi), it works.

apparatus · October 21, 2024, 8:47am

Yes, it works for me with 500 PPI. At least I didn’t have the error while converting the PDF file up to page 65.
Maybe the issue is related to the memory balancing because I’ve noticed that xen-balloon process was using a lot of CPU resources.
Maybe there is some race condition when you don’t have enough free memory for the memory balancer to assign to your disposable qube (or the memory assignment is not fast enough) and the conversion operation fails because of this.

Try to disable the memory balancing and set the initial memory to 2 GB in the disposable template that you use for the PDF conversion for a test.

shimmy_jimmy · October 21, 2024, 9:58am

It’s not working with 500 PPI even with 2.5 GB assigned to qube from which I convert and to qube where it’s converting. But I noticed that there was an update and now default is 300 PPI and it works with that value even without disabled memory balancing. I’m pretty satisfied with 300 PPI. Maybe I just have too low memory for that, idk.

alimirjamali · October 21, 2024, 11:22am

Glad that you liked it. There is also the new --resolution for qvm-convert-pdf.

The limit is not memory. It is /tmp size. You may try to increase it to something over the default 1GB and try again.

p.s. /tmp is used to temporary store uncompressed PDF pages during conversion. It is also used for temporary storage of uncompressed images during qvm-convert-image operation.