FeedReader (FeedReader v0.1.0)

Copy Markdown

Standalone feed fetching and parsing library.

This module exposes format detection, validation, local parsing, and streaming helpers. Remote sessions download and checksum into a temporary file before exposing their one-pass event stream.

Summary

Types

format_id()

:rss, :atom, :rdf, :unknown, or a custom format's atom ID.

Functions

close(session)

Closes an owned remote session; safe after completion or repeated calls.

detect_format(content, opts \\ [])

Detects a format from a bounded binary prefix.

fetch_and_parse(url, opts \\ [])

Downloads and collects a feed, requiring a finite positive :max_items.

parse_binary(content, opts \\ [])

Collects XML binary content into a bounded parse result.

parse_file(path, opts \\ [])

Collects a caller-owned local XML file into a bounded parse result.

parse_stream(chunks, opts \\ [])

Collects an enumerable of binary XML chunks into a bounded parse result.

stream(url, opts \\ [])

Downloads and checksums a feed before returning a lazy event session.

stream_file(path, opts \\ [])

Lazily parses a caller-owned local XML file in one SAX pass.

stream_file_verified(path, opts \\ [])

Validates the complete caller-owned local feed before returning an event session.

stream_verified(url, opts \\ [])

Downloads and validates a feed before exposing its remote event session.

validate(content, format, opts \\ [])

Checks only the bounded-prefix signature for a format ID or module.

Types

format_id()

@type format_id() :: atom()

:rss, :atom, :rdf, :unknown, or a custom format's atom ID.

Functions

close(session)

@spec close(FeedReader.Session.t()) :: :ok

Closes an owned remote session; safe after completion or repeated calls.

detect_format(content, opts \\ [])

@spec detect_format(binary(), keyword()) :: format_id()

Detects a format from a bounded binary prefix.

Options

  • :max_detection_bytes — positive byte limit for the inspected prefix; defaults to 65536. Invalid values yield :unknown.
  • :formats — custom format modules checked after RSS, Atom, and RDF in list order. Duplicates are removed. Modules provide matches_prefix?/1 and id/0.

fetch_and_parse(url, opts \\ [])

@spec fetch_and_parse(String.t(), keyword()) ::
  {:ok, FeedReader.RemoteResult.t() | :not_changed | :not_modified}
  | {:error, term()}

Downloads and collects a feed, requiring a finite positive :max_items.

Memory use is proportional to the number and size of collected items. The collector scans through the terminal parser event and returns no partial result on an item-limit breach or parse error.

Options

  • :http_client — FeedReader.Transport implementation; defaults to FeedReader.ReqTransport.

  • :http_options — Req options forwarded to Req.run/2. The default transport forces raw streaming and disables automatic decompression. :headers inside this list replaces the default user-agent and accepted-compression headers.

  • :previous_etag — binary ETag sent verbatim as If-None-Match on GET; a 304 returns :not_modified without downloading a body.

  • :http_retry_attempts — non-negative retries after the initial GET; defaults to 5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.

  • :retry_delay — non-negative milliseconds or an arity-one delay function; defaults to 0. Invalid values use the default.

  • :previous_checksum — decoded-body checksum; a match returns :not_changed and removes the downloaded file.

  • :max_compressed_bytes, :max_decoded_bytes, and :max_temp_bytes — non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are 52428800 bytes each.

  • :max_expansion_ratio — non-negative decoded-to-compressed byte ratio; defaults to 100. Invalid limits use their defaults. Checksums are computed over decoded bytes.

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — required positive limit for accepted items; exceeding it returns {:error, {:max_items_exceeded, max_items}} without partial data.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

parse_binary(content, opts \\ [])

@spec parse_binary(binary(), keyword()) ::
  {:ok, FeedReader.ParseResult.t()} | {:error, term()}

Collects XML binary content into a bounded parse result.

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — required positive limit for accepted items; exceeding it returns {:error, {:max_items_exceeded, max_items}} without partial data.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

parse_file(path, opts \\ [])

@spec parse_file(Path.t(), keyword()) ::
  {:ok, FeedReader.ParseResult.t()} | {:error, term()}

Collects a caller-owned local XML file into a bounded parse result.

The parser stops after the feed's root element, ignoring content appended after a complete feed. An item accepted beyond the limit returns {:error, {:max_items_exceeded, max_items}} without partial data.

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — required positive limit for accepted items; exceeding it returns {:error, {:max_items_exceeded, max_items}} without partial data.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

parse_stream(chunks, opts \\ [])

@spec parse_stream(Enumerable.t(), keyword()) ::
  {:ok, FeedReader.ParseResult.t()} | {:error, term()}

Collects an enumerable of binary XML chunks into a bounded parse result.

Chunks are written incrementally to a temporary file and then parsed through the same event stream used by parse_file/2. The temporary file is removed on success and failure.

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

  • :max_items — required positive limit for accepted items; exceeding it returns {:error, {:max_items_exceeded, max_items}} without partial data.

  • :item_filter — optional one-argument function applied before counting and retaining items; invalid values return {:error, {:invalid_option, :item_filter}}.

stream(url, opts \\ [])

@spec stream(String.t(), keyword()) ::
  {:ok, FeedReader.Session.t() | :not_changed | :not_modified}
  | {:error, term()}

Downloads and checksums a feed before returning a lazy event session.

The session emits feed metadata followed by items from one local XML pass. Call close/1 in an after block if enumeration may stop early. Matching previous checksums or a conditional HTTP 304 return :not_changed or :not_modified without a session.

Options

  • :http_client — FeedReader.Transport implementation; defaults to FeedReader.ReqTransport.

  • :http_options — Req options forwarded to Req.run/2. The default transport forces raw streaming and disables automatic decompression. :headers inside this list replaces the default user-agent and accepted-compression headers.

  • :previous_etag — binary ETag sent verbatim as If-None-Match on GET; a 304 returns :not_modified without downloading a body.

  • :http_retry_attempts — non-negative retries after the initial GET; defaults to 5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.

  • :retry_delay — non-negative milliseconds or an arity-one delay function; defaults to 0. Invalid values use the default.

  • :previous_checksum — decoded-body checksum; a match returns :not_changed and removes the downloaded file.

  • :max_compressed_bytes, :max_decoded_bytes, and :max_temp_bytes — non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are 52428800 bytes each.

  • :max_expansion_ratio — non-negative decoded-to-compressed byte ratio; defaults to 100. Invalid limits use their defaults. Checksums are computed over decoded bytes.

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

stream_file(path, opts \\ [])

@spec stream_file(Path.t(), keyword()) ::
  {:ok, FeedReader.Session.t()} | {:error, term()}

Lazily parses a caller-owned local XML file in one SAX pass.

The returned session emits feed metadata before normalized items. The file is never removed by the library, including when enumeration stops early.

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.
  • :formats — additional format modules, checked after RSS, Atom, and RDF.
  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.
  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.
  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.
  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.
  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.
  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

stream_file_verified(path, opts \\ [])

@spec stream_file_verified(Path.t(), keyword()) ::
  {:ok, FeedReader.Session.t()} | {:error, term()}

Validates the complete caller-owned local feed before returning an event session.

The source file is never removed. :max_parse_passes bounds validation and event delivery passes.

Options

  • :format — format ID or FeedReader.Format module; defaults to detection.
  • :formats — additional format modules, checked after RSS, Atom, and RDF.
  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.
  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.
  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.
  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.
  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.
  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

stream_verified(url, opts \\ [])

@spec stream_verified(String.t(), keyword()) ::
  {:ok, FeedReader.Session.t() | :not_changed | :not_modified}
  | {:error, term()}

Downloads and validates a feed before exposing its remote event session.

Matching checksums and conditional HTTP 304 responses return :not_changed and :not_modified without a session. Call close/1 if enumeration may stop early.

Options

  • :http_client — FeedReader.Transport implementation; defaults to FeedReader.ReqTransport.

  • :http_options — Req options forwarded to Req.run/2. The default transport forces raw streaming and disables automatic decompression. :headers inside this list replaces the default user-agent and accepted-compression headers.

  • :previous_etag — binary ETag sent verbatim as If-None-Match on GET; a 304 returns :not_modified without downloading a body.

  • :http_retry_attempts — non-negative retries after the initial GET; defaults to 5. The Req transport retries status 400 and 500 and most transport errors; TLS and redirect failures are terminal. Redirects are limited to five.

  • :retry_delay — non-negative milliseconds or an arity-one delay function; defaults to 0. Invalid values use the default.

  • :previous_checksum — decoded-body checksum; a match returns :not_changed and removes the downloaded file.

  • :max_compressed_bytes, :max_decoded_bytes, and :max_temp_bytes — non-negative raw-body, decoded-body, and temporary-file byte limits; defaults are 52428800 bytes each.

  • :max_expansion_ratio — non-negative decoded-to-compressed byte ratio; defaults to 100. Invalid limits use their defaults. Checksums are computed over decoded bytes.

  • :format — format ID or FeedReader.Format module; defaults to detection.

  • :formats — additional format modules, checked after RSS, Atom, and RDF.

  • :chunk_size — positive XML input chunk size; defaults to 65536 bytes.

  • :max_detection_bytes — positive detection prefix limit; defaults to 65536 bytes. Invalid values disable detection, so provide a valid explicit :format to parse in that case.

  • :feed_url — source URL supplied to normalization; defaults to nil for local sources and the final response URL for remote sources.

  • :max_xml_events, :max_xml_depth, :max_attributes_per_element, :max_attribute_bytes, :max_feed_metadata_bytes, and :max_item_bytes — non-negative SAX and retained-data limits. Defaults are respectively 1000000, 256, 128, 65536, 1048576, and 4194304. Zero is valid; invalid values use the defaults.

  • :max_queue_bytes — non-negative maximum serialized size of one pending event; defaults to :max_item_bytes. Zero is valid; invalid values use that default.

  • Verified APIs also accept :max_parse_passes (default 5), counting validation and delivery.

validate(content, format, opts \\ [])

@spec validate(binary(), atom() | module(), keyword()) :: boolean()

Checks only the bounded-prefix signature for a format ID or module.

Options

  • :max_detection_bytes — positive byte limit for the inspected prefix; defaults to 65536. Invalid values yield :unknown.
  • :formats — custom format modules checked after RSS, Atom, and RDF in list order. Duplicates are removed. Modules provide matches_prefix?/1 and id/0.