jhnUdZddlmZddlZddlZddlZddlZddlmZm Z ddl m Z dZ dddd d d d d d Z e ejjde ej#j$dz Zd:dZd;dZdd!Z#eddd=d"Z$d?d#Z%d$Z&ed%d@d&Z'ed%d@d'Z(gd(Z)d)e*d*<d+Z+d,Z,dd-dAd.Z-dd-dBd/Z.d;d0Z/dBd1Z0d2dd3 dCd4Z1dDd5Z2dBd6Z3dEd7Z4dFd8Z5dGd9Z6y)Ha)OCR engine for the Togen OCR Tool (DVI-490, plan DVI-463). Wraps OCRmyPDF to turn an image-only / scanned PDF into a text-selectable ("searchable") PDF and emit the recognized text as a sidecar. Board directive (D1): keep this a clean seam so a cloud fallback for low-confidence pages can be added later without route/UI rework. The engine returns an `OcrResult` carrying the extracted text plus a per-page metadata list; a future cloud-fallback pass can inspect `pages[*].confidence` and re-OCR only the weak pages, then the route/UI stay unchanged. ) annotationsN) dataclassfield)PathlocalLocalznOn-device OCR (Tesseract) with region heuristics. Runs entirely on the Togen host; no data leaves the network.)idlabel descriptionclaudezClaude Vision (Anthropic API)zUses Claude's vision model to read text from scanned drawings. Requires an Anthropic API key configured in Assistant settings. Only available when the Assistant feature is enabled and accessible to users (or for admins).)rr OCR_SETTINGS_FILEzocr_settings.jsoncbtjDcgc] }t|c}Scc}w)zDReturn the registered backends as a list of public dicts (UI-ready).) OCR_BACKENDSvaluesdict)bs !/var/www/html/togen/ocr_engine.py list_backendsr:s#)002 3DG 33 3s,c2tjrZ tjtj }t |t r|jdtvr|dStStS#tjtf$rYtSwxYw)z>Return the configured backend id, falling back to the default.backend) r is_filejsonloads read_text isinstancergetrJSONDecodeErrorOSErrorDEFAULT_OCR_BACKEND)datas r load_backendr!?s  " ::/99;Persist the selected backend id. Raises ValueError if unknown.zunknown OCR backend r)indentN)r ValueErrorr write_textrdumpsrs r save_backendr)Ks;l"/{;<<  Y,@!KLz skip-textzredo-ocrz force-ocrc*eZdZUdZded<dZded<y)PageInfozPer-page OCR metadata. `confidence` is the seam for a cloud fallback: OCRmyPDF/Tesseract does not surface a per-page confidence through the public API, so it stays None here until a richer engine populates it.intnumberNz float | None confidence)__name__ __module__ __qualname____doc____annotations__r/r*rr,r,\sM K#J #r*r,c8eZdZUdZded<eeZded<y) OcrResultstrtext)default_factoryzlist[PageInfo]pagesN)r0r1r2r:r4rlistr<r5r*rr7r7fsD#N!$7E>7r*r7ceZdZdZy)OcrEngineUnavailablezGRaised when the ocrmypdf package (or its host tools) are not installed.N)r0r1r2r3r5r*rr?r?lsQr*r?modercx| t}|tvrt}|dk(rt|||St |||S)aaOCR `in_path` into a searchable PDF at `out_path`. Dispatches to the configured backend (defaults to the admin-selected one in OCR_SETTINGS_FILE). Returns an OcrResult whose `text` is the recognized sidecar text and whose `pages` carries per-page metadata. Raises OcrEngineUnavailable if the backend's host tools are not installed. r rA)r!rr_ocr_pdf_claude_ocr_pdf_local)in_pathout_pathrArs rocr_pdfrHpsC.l"%(wt<< '8$ 77r*>.bmp.gif.jpg.png.tif.jpeg.tiff.webp> .doc.odp.ods.odt.ppt.rtf.xls.docx.pptx.xlsx.pdfcftt|jjtvS)zDTrue if the filename's extension is something ocr_file() can handle.)rr9suffixlowerPROCESSABLE_EXTS)filenames ris_processableras& H  % % + + -1A AAr*c2t|}t|}|jj}|dk(rt||||S|tvrKt j 5}t|dz }t||t||t|cdddS|tvrOt j 5}t|t|}| tdt||||cdddStd|xsd#1swYrxYw#1swY'xYw)aOCR any supported file into a searchable PDF at `out_path`. PDFs go straight to ocr_pdf(). Images are wrapped into a one-page PDF first (and forced through OCR, since an image has no text layer). Office documents are converted to PDF via LibreOffice if it is installed on the host; otherwise OcrEngineUnavailable is raised with guidance. Unknown extensions raise ValueError.r[r@z input.pdfNzConverting Office documents to PDF requires LibreOffice (soffice) on the Togen host. Convert the file to PDF first, or ask an admin to install LibreOffice.zUnsupported file type for OCR: z(none)) rr]r^rH IMAGE_EXTStempfileTemporaryDirectory _image_to_pdf FORCE_OCR OFFICE_EXTS_office_to_pdfr?r%)rFrGrArexttdpdf_ins rocr_filerms#7mGH~H ..   C f}wtWEE j  ( ( * Nb"X +F '6 *68)WM N N k  ( ( * Ib#GT"X6F~*> 68$H I I 6sh6GH II N N  I Is!-D41D D  Dc ddl}|jt |} |j }|j t|j|y#t$r}td|d}~wwxYw#|j wxYw)z9Wrap a raster image into a single-page PDF using PyMuPDF.rNz.PyMuPDF (fitz) is required to OCR image files.) fitz ImportErrorr?openr9convert_to_pdfcloser write_bytes)img_pathpdf_outroexcdoc pdf_bytess rrfrfs} ))CM "C&&(  Mi( " <   s"AA9 A6% A11A69B c dddl}ddl}|jdxs|jd}|sy |j|ddddt |t |gd d d t|t|jd zz }|jr|SdS#|j t f$rYywxYw) zConvert an Office document to PDF via headless LibreOffice. Returns the output Path, or None if LibreOffice is unavailable / conversion failed.rNsoffice libreofficez --headlessz --convert-topdfz--outdirT)checkcapture_outputtimeoutr[) shutil subprocesswhichrunr9SubprocessErrorrrstemexists)srcwork_dir_shutilrr{outs rrirismmI&F'-- *FG  lNE XC 2tS   x.DINNV3 4C**,3(D(  & & 0s.BB/.B/rCcJ|tvrtdtd| ddl}t |}|j d}tdditd ditd dii|}|jt|t|ft|d d |d }|jr|jdd}|r|jdng}tt!|D cgc]} t#| dz} } t%|| S#t$r}t d|d}~wwxYwcc} w)z2Local (Tesseract/OCRmyPDF) backend implementation.zmode must be one of z, got rNzocrmypdf is not installed. The OCR host packages (ocrmypdf, tesseract, ghostscript, qpdf) are provisioned by the Linux Automation Engineer subtask (DVI-489).z.txt skip_textTredo_ocr force_ocrF)sidecar progress_barr8zutf-8replace)encodingerrors r.r:r<) VALID_MODESr%ocrmypdfrpr?r with_suffix SKIP_TEXTREDO_OCRrgocrr9rrsplitrangelenr,r7) rFrGrArrw sidecar_pathflagsr: page_textsir<s rrErEsN ;/ }F4(KLLH~H''/L K&:t$K&   E HLL G  H L!    D%%wy%I&*D!rJ-23z?-C DXQU # DE D $e ,,C " ;   > EsDD  D DDc ddlm}m}m} ddl}t|}t|} |jt|}g} g} tt|D]} |j| } |jdd} | j| }|j!d } ||d g}|j#d d }| j%|| j%t'| d z |j)ddl}|j-t|t|dj/| }t1|| S#t$r}t d|d}~wwxYw#t$r}t d|d}~wwxYw#t$r}t d||d}~wwxYw#||f$r}t d | d zd||d}~wwxYw#|j)wxYw)a#Claude-vision OCR backend: rasterize each PDF page and send to Claude. Requires the org-level Anthropic API key configured in assistant_settings.json (via the Assistant admin settings). Falls back gracefully when the key is absent or the anthropic package is not installed. r)extract_fieldsAssistantUnavailableAssistantErroru@assistant module not found — cannot use Claude-vision backend.NuQPyMuPDF (fitz) is not installed — required for Claude-vision OCR rasterization.z*Failed to open PDF for Claude-vision OCR: g@)matrixpng full_textr8z!Claude-vision OCR failed on page rz: rrr) assistantrrrrpr?rorrqr9 Exceptionrr load_pageMatrix get_pixmaptobytesrappendr,rsrcopy2joinr7)rFrGrArrrrwrorxall_textr<page_numpagematpix img_bytesresult page_textrcombineds rrDrDsRR  7mGH~H`iiG %HEc#h 8H==*D++h1C///-C E*I ' M#JJ{B7 OOI & LLA6 7# 8&  LLWs8}-yy"H (% 00g " N   " _   `"%OPSu#UV\__`&).9 *71 ~RuM  sv E E&FAG%F#1G E# EE#& F/ E;;F F  FF #G*F??GGG)') customer_namez customer name)rcustomer)rzgeneral contractor)rzcontractor name)r contractor)rclient)rzsold to) project_namez project name)rz project title)rproject) job_numberz job number)rzjob no)rzjob #) product_codez product code)rz catalog no)rzmodel no)rproduct)part_refzpart reference)rzpart ref)rz part number)rzpart no)rz piece mark)rmark)weightz unit weight)rr) structure_idz structure id)rz assembly id)r structure)rassembly) spec_typez spec./type)rz spec type)r specification)rspec)structure_sizezstructure size)rsize)stationr)rsta)production_datezproduction date)rdatezlist[tuple[str, str]]_FIELD_ALIASES?r(cd} t|}t|}|r|S t||}t|S#t$rd}Y4wxYw#t$rd}Yt|SwxYw)uRead labeled metadata from a PDF's title block and map it to Create dialog fields. Returns a dict of {field: value} for whatever was found (possibly empty). Never raises for a malformed/unreadable PDF — extraction is best-effort prefill, so failures degrade to an empty result. Strategy: prefer the embedded text layer (vector drawings / carrier sheets), restricted to the title-block region for region heuristics. If that yields nothing (image-only/scanned PDF), fall back to OCR via the selected backend. r8)_titleblock_text_embeddedr_parse_labeled_fields_titleblock_text_ocr)pdf_pathrr:fieldsocr_texts rextract_title_block_fieldsrs D(2#4 (F  '': ! **    **s 6 A AA AAc t|}|r|jr|S t||xsdS#t$rd}Y2wxYw#t$rYywxYw)u3Return the raw title-block text (embedded layer, else OCR fallback) that feeds extract_title_block_fields. Exposed for callers that need to scan the text directly rather than rely on labeled "LABEL: value" pairs — e.g. the Project Builder tranId discovery (DVI-661). Best-effort: never raises.r8)rrstripr)rrr:s rextract_title_block_textrsc (2   #Hg6<"<  s 2A AA AAc 4ddlm}|t|}|jsy|jd}|j}t |j xsd}t |jxsd}|tz|tzgfd}|j|syjdg}g} d } d } D]\} } }}td d t|| z}| t| | z |kr&| j| |f| | n| } t| |} Z|jd jdt!| D| |fg} | } |} | r0|jd jdt!| Ddj|S)zExtract embedded text from the title-block region of page 1, preserving rough line structure. Returns "" if the page has no usable text layer.r) PdfReaderr8rc |r|jsy|d|d}}|k\r)| kr#j|||t|xsdfyyy)Nr)rrfloat) r:cmtm font_dict font_sizexychunksx_miny_maxs r_visitorz+_titleblock_text_embedded.._visitorsU4::< !ube1 :!u* MM1auY-=#'>? @%:r*) visitor_textc|d |dfS)Nrrr5)cs rz+_titleblock_text_embedded..s!uadmr*)keyNr@g? c3&K|] \}}| ywNr5.0_xts r z,_titleblock_text_embedded..s!=A!!=c3&K|] \}}| ywrr5rs rrz,_titleblock_text_embedded..s9EBa9r )pypdfrr9r<mediaboxrwidthheight_TITLEBLOCK_X_FRAC_TITLEBLOCK_Y_FRAC extract_textsortmaxabsrrsorted)rrreaderrboxr r rlinescurcur_ycur_fsrrrfstolrrrs @@@rrrs s8} %F << <<?D --C #))  #E 3::  %#F & &E ' 'E46FA 8,  KK+K,E#%CE F  1a#sSV_,- =CE Nc1 JJ1v AEE_F LL!=!== >q6(CEF   SXX9VC[99: 99U r*c~ ddlm}m}t j 5}t |} |t|}|js dddy|}|j|jd|dz }t|d5} |j| ddd|dz } t|| t| t!| cdddS#t$rYywxYw#1swYFxYw#t$rt |}Y_wxYw#t$r YdddywxYw#t$r YdddywxYw#1swYyxYw) aOCR fallback for scanned/image-only PDFs: OCR page 1 via the configured backend, then re-read the *coordinate-aware* text from the resulting searchable PDF. Returns "" if OCR is unavailable. We deliberately do NOT use the linear OCR sidecar text here: title blocks are laid out as two columns (labels left, values right), and Tesseract's page segmentation emits the whole label column and then the whole value column, so "Customer Name:" and its value land on different lines and never pair up (DVI-504). Reading the OCR'd PDF's embedded text layer restores per-word positions, so _titleblock_text_embedded groups each label with its value by row and the same-line "LABEL: value" parser works.r)r PdfWriterr8Nz page1.pdfwbzocr.pdfr@)rrrrprdrerr9r<add_pagerqwriterrHrgr?r) rrrrrktd_pathrwriterrFfhrGs rrrsI.  $ $ &"r( %s8}-F<< [F OOFLLO , +Ggt$ ! R  ! Y&  GXIw G ,X6+  ! ! %8nG % $ '$  /, -sC D3C+6C+CC+"D3(D< D CCC( $C++D?D3DD3 DD3DD3 D0%D3/D00D33D<F)rawrcNt||||}|r t|St|S)u7OCR a user-selected sub-region of one page (DVI-645). `frac_rect` is (x0, y0, x1, y1) as fractions (0..1) of the page's rendered width/height — the front-end maps the on-screen selection to fractions so it never needs to know the server render zoom. Strategy mirrors extract_title_block_fields: prefer embedded text inside the rectangle (instant, works for vector drawings / carrier sheets); if the region has no text layer (scanned/image PDF), render just the clip and OCR it. Never raises — region OCR is convenience and degrades to empty output. Returns the raw recognized text (str) when `raw` is True — used to fill one targeted field. Otherwise returns a {field: value} dict parsed from "LABEL: value" lines — used for the "OCR this region → best guess" flow. ) _region_text_clean_region_textr)rpage_no frac_rectr"rr:s rextract_regionr(s-" '9g >D !$''  &&r*c2 ddl} |\}}}} |j t |} |dks|| j k\r | jy| j|} | j} |j| j|| jzz| j|| jzz| j|| jzz| j|| jzz} | jd| xsdj!} | r| | jSt#| | || jS#t$rYywxYw#ttf$rYywxYw#t$rYywxYw#t$rd} YqwxYw#| jwxYw)uKEmbedded-text-in-rect → OCR-the-clip fallback. Returns "" on any failure.rNr8r:)clip)ror TypeErrorr%rqr9 page_countrsrrectRectx0r y0r get_textr_region_ocr_text)rr&r'rrofx0fy0fx1fy1rxrrr*embeddeds rr$r$'s}&S#siiH & Q;'S^^3 }}W% IIyy DD3= !$$qxx"7 DD3= !$$qxx"7   f4 8>BEEGH   dG4 7  z "  H  soEEE$FBF*%E3F# F E  E E! E!$ E0/E03 F>FFFFc ddl} |j|jdd|}t j 5}t |}|dz }|dz } |j|jdd } | j} | j|j|  t||t|  |jt|} | j!dj#d xsdj%| jcdddS#t$rYywxYw#t$rYywxYw#| jwxYw#t$r YdddywxYw#t$r Ydddyt$r YdddywxYw#| jwxYw#t$r YdddywxYw#1swYyxYw) uRender the clip to an image-only PDF and OCR it, then read the coordinate-aware text from the result (the linear sidecar jumbles two-column title blocks — see _titleblock_text_ocr). Returns "" if OCR is unavailable.rNr8r)rr*z region.pdfzregion_ocr.pdfr)streamfiletyper@r:)rorrrrdrerrqrrrrsrtrHrgr?r9rr1r) rr*rrorrkrrFrGimg_docryods rr2r2IsooT[[c%:oF  $ $ &"r(L(-- iis{{5'9EiJG #224      *  GXIw G  3x=)B  Q008>BEEG 1     $ !" %"   52 3sD$#D3G"E8E!E*E.>F,1F F,$ D0/D03 D?>D?EE E+ G*E++G. F7GF GFGF))F,, G5G?GGG cR|sytjdd|jS)zwCollapse a region's recognized text into a single trimmed line, suitable for dropping straight into one form field.r8\s+r)resubr)r:s rr%r%rs&  66&#t $ * * ,,r*cL|siSttdd}i}|jD]}tjd|Dcgc]}|j }}t |D]\}}|rd|vrd|vr|D]\}} ||vr tjtj| dz|tj} | sIt| jd} | s0|dzt|kr||dznd } | rd| vrd| vr t| } | r| ||<|Scc}w) aParse "LABEL: value" pairs out of title-block text into Create fields. Splits each line into column segments on runs of 2+ spaces/tabs (title blocks are grid-like), then matches the longest known label that a segment starts with. First non-empty value per field wins.ct|dS)Nr)r)fas rrz'_parse_labeled_fields..s#be*r*T)rreversez \s{2,}|\t+:#z\s*[:#]+\s*(.*)$rr8) rr splitlinesr@rr enumeratematchescape IGNORECASE _clean_valuegroupr) r:pairsrraw_linessegsrseg fieldnamealiasmvalnxts rrrzs7  >'cn$)  5#HHRYYu-0CCS"--X"1771:. *+QT):$q1u+Cs#~#S.*3/%(C N%  0 J/FsD!ctjdd|jdj}|jdvry|ddS)zTidy a captured value: collapse whitespace, strip stray separators, cap length. Drops values that are obviously empty placeholders.r?rz :#-_/|)r8zn/ananonetbd-r8Nx)r@rArr^)rWs rrMrMsH &&c " ( ( 5 ; ; =C yy{;; t9r*)returnz list[dict])r_r9)rr9r_None)rAr9r str | Noner_r7)r_bool)r_r`)rAr9r_r7)rrar_dict[str, str])rrar_r9)r&r-r"rbrra)r&r-rrar_r9)r:r9r_r9)r:r9r_rc)rWr9r_r9)7r3 __future__rrosr@rd dataclassesrrpathlibrrrenvironr__file__resolveparentr rr!r)rrrgrr,r7 RuntimeErrorr?rHrcrhr_rarmrfrirErDrr4r r rrrrr(r$r2r%rrMr5r*rrms # ( = 0 (  ,JJNNT(^335< ) )*6?(-V7@<1R()%(bCG+6AE"7t(X %D''/9'.D&R-#Lr*