Which RFC(s) for media type should we refer to?
MURATA Makoto
eb2m-mrt at asahi-net.or.jp
Thu Dec 11 15:18:41 CET 2014
Folks,
As for the definition of media types, I have carefully
compared RFC 7230 and opc-contentTypes.xsd
(a schema of OPC). This mail covers all my findings about this
topic.
Summary: The regular expression in opc-contentTypes.xsd has
to be chnaged.
The opc-contentTypes.xsd XSD schema has an ugly regular expression for
defining permissible values of the attribute @ContentType. This
regular expression can be significantly simplified by removing
unnecessary parentheses and defining two parsed entities:
<!ENTITY X "[\p{IsBasicLatin}-[\p{Cc}\(\)<>@
,;:\\"/\[\]\?=\{\}\s\t]]">
<!ENTITY Y
"[\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]">
The result is
"$X+/$X+(\s*;\s*($X+=(($X+)|("(&Y;|\s+|(\\[\p{IsBasicLatin}]))*"))))*"
We study X, Y, and the resulting regular expression in this sequence.
1) X
I believe that X precisely captures
token = 1*tchar
tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*"
/ "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
/ DIGIT / ALPHA
; any VCHAR, except delimiters
in RFC 7230. I created a list of basic latin characters and removed
all that match [\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t].
The result is exactly the same as tchar shown above.
2) Y
Y is similar to X, but there are two important differences:
First, any Latin-1 Supplement character is allowed if it is not not a
control chrar (u+0080 - u+008F).
Second, [\(\)<>@,;:\\/\[\]\?=\{\}\s\t] is not excluded. This
is because quoted strings are allowed as part of doubly quoted strings.
Third, [\n\r] is excluded. This looks unnecessary, since \p{Cc}
is already excluded.
3)
$X+/$X+(\s*;\s*($X+=(($X+)|("(&Y;|\s+|(\\[\p{IsBasicLatin}]))*"))))*
Here "$X+/$X+" means a media type name such as text/html. The rest is
a collection of parameters, delimited by ";" (possibly having leading
and trailing whitespace). A parameter name followed by "=" is
captured by $X+=.
A parameter value is captured by
(($X+)|("(&Y;|\s+|(\\[\p{IsBasicLatin}]))*"))
This allows either a name-like string ($X+) or a doubly-quoted string.
A doubly quoted string consists of a sequence of (i) what matches &Y;,
(ii) a sequence of whitespace characters, or (iii) the backslash
character followed by a basic latin character.
4) Comparison with RFC 7230
Is the regular expression consistent with RFC 7230? The only
questionable part is the definition of doubly-quoted strings.
RFC 7230 has rules for defining the syntax of doubly quoted strings.
quoted-string = DQUOTE *( qdtext / quoted-pair ) DQUOTE
qdtext = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text
obs-text = %x80-FF
First, RFC 7230 allows HTAB as part of a doubly quoted string. Our
regular expression also allows HTAB, since \s+ matches HTAB.
Second, RFC 7230 does not allow LF or NL. But \s+ does. To precisely
capture RFC 7230, our regular expression should use \t rather than \s.
Third, RFC 7230 disallows the backslash character as part of a doubly
quoted string, but the regular expression in opc-contentTypes.xsd does
not. Here RFC 7230 looks reasonable, since the backslash character is
used for character escaping. I believe that we have to fix the
regular expression.
Fourth, RFC 7230 allows %x80-9F but the regular expression does not.
(XML 1.0 allows the use of these control characters.) But the regular
expression in opc-contentTypes.xsd does not.
Fifth, \\[\p{IsBasicLatin}] in the regular expression allows every
basic latin character to follow the backslash character and nothing
more. However,
quoted-pair = "\" ( HTAB / SP / VCHAR / obs-text )
in RFC 7230 is different in two points:
I: RFC 7230 alows any Latin-1 Supplement character
to follow the slash character.
II: RFC 7230 does not allow invisible ASCII characters
(except HTAB and SP) to follow the slash character.
This might not be an issue, since XML 1.0 does not allow
the use of C0 control characters except HTAB, NL, and CR.
Here is my proposed rewrite of the definition of Y and the entire
regular expression. It covers all the points except point II,
<!ENTITY Y
"[\p{IsBasicLatin}-[\p{Cc}\\"]]|[\p{IsLatin-1Supplement}\t]">
"$X+/$X+(\s*;\s*($X+=(($X+)|("(&Y;|(\\[\p{IsBasicLatin}\p{IsLatin-1Supplement}))*"))))*"
Regards,
Makoto
2014-11-02 22:33 GMT+09:00 MURATA Makoto <eb2m-mrt at asahi-net.or.jp>:
> Folks,
>
> The regular expression in opc-contentTypes.xsd has another interesting
> subexpression:
>
>
> ("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"
>
> This subexpression matches a doubly-quoted string. But what character
> is allowed as a part of this doubly-quoted string? Is this
> subexpression consistent with RFC 7230?
>
> First, unlike the first regular expression,
> neither \(\)<>@,;:\\ nor /\[\]\?=\{\}\s\t are excluded.
> This is because RFCs 2045 and 7230 allow tspecials as part of
> doubly-quoted strings. This is nice.
>
> Second, escaped characters such as \a are always allows by
> (\\[\p{IsBasicLatin}]) as long as escaped characters are from #x0000
> to #x007F. RFC 7230 allows more characters (#x0080 to #x007F) to be
> escaped, but does not allow invisible characters such as the space
> character to be escaped. Thus, there is a discrepancy here.
>
> Third, \p{IsLatin-1Supplement} represents characters
> from #x0080 to #x00FF. See
> http://www.w3.org/TR/xmlschema-2/#nt-charClassEsc
> This matches obs-text in RFC 7230.
>
> Fourth, RFC 7230 (to be precise, qdtext) does not allow REVERSE
> SOLIDUS, but the subexpression does. Thus, we have another
> discrepancy.
>
> qdtext = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text
>
> Regards,
> Makoto
>
> 2014-11-02 16:56 GMT+09:00 MURATA Makoto <eb2m-mrt at asahi-net.or.jp>:
>
>> Oops, I forgot to point out that \{ and \} are disallowed by
>> our regular expression, but they are not tspecials as
>> specified in RFC 2045. RFC 7230 does
>> not allow these two characters, and exactly catches
>> the enumerated list in my previous mail.
>>
>> https://tools.ietf.org/html/rfc7230#section-3.2.6
>>
>> One could say that our regular expression is already
>> aligned with RFC 7230 rather than RFC 2045.
>>
>> Regards,
>> Makoto
>>
>> 2014-11-02 16:15 GMT+09:00 MURATA Makoto <eb2m-mrt at asahi-net.or.jp>:
>>
>>> Folks,
>>>
>>> I pointed out that
>>>
>>> [\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\
>>> [\]\?=\{\}\s\t]]
>>>
>>> is used repeatedly. This appears to represent characters in
>>>
>>> token := 1*<any (US-ASCII) CHAR except SPACE, CTLs,
>>> or tspecials>
>>>
>>> where
>>>
>>>
>>> tspecials := "(" / ")" / "<" / ">" / "@" /
>>> "," / ";" / ":" / "\" / <">
>>> "/" / "[" / "]" / "?" / "="
>>> ; Must be in quoted-string,
>>> ; to use within parameter values" /
>>>
>>> They both represent any of the following characters:
>>>
>>> - 0021;EXCLAMATION MARK;Po;0;ON;;;;;N;;;;;
>>> - 0023;NUMBER SIGN;Po;0;ET;;;;;N;;;;;
>>> - 0024;DOLLAR SIGN;Sc;0;ET;;;;;N;;;;;
>>> - 0025;PERCENT SIGN;Po;0;ET;;;;;N;;;;;
>>> - 0026;AMPERSAND;Po;0;ON;;;;;N;;;;;
>>> - 0027;APOSTROPHE;Po;0;ON;;;;;N;APOSTROPHE-QUOTE;;;;
>>> - 002A;ASTERISK;Po;0;ON;;;;;N;;;;;
>>> - 002B;PLUS SIGN;Sm;0;ES;;;;;N;;;;;
>>> - 002D;HYPHEN-MINUS;Pd;0;ES;;;;;N;;;;;
>>> - 002E;FULL STOP;Po;0;CS;;;;;N;PERIOD;;;;
>>> - 0-9
>>> - A-Z
>>> - 005E;CIRCUMFLEX ACCENT;Sk;0;ON;;;;;N;SPACING CIRCUMFLEX;;;;
>>> - 005F;LOW LINE;Pc;0;ON;;;;;N;SPACING UNDERSCORE;;;;
>>> - 0060;GRAVE ACCENT;Sk;0;ON;;;;;N;SPACING GRAVE;;;;
>>> - a-z
>>> - 007C;VERTICAL LINE;Sm;0;ON;;;;;N;VERTICAL BAR;;;;
>>> - 007E;TILDE;Sm;0;ON;;;;;N;;;;;
>>>
>>>
>>> The regular expression allow any of these characters as
>>> part of a top-level media type name, second-level
>>> media type name, and parameter name,
>>>
>>> Regards,
>>> Makoto
>>>
>>> 2014-10-25 22:08 GMT+09:00 MURATA Makoto <eb2m-mrt at asahi-net.or.jp>:
>>>
>>>> Caroline,
>>>>
>>>> Thank you for your through study! This is an
>>>> eye opener.
>>>>
>>>> Both RFC 2616 and RFC 7321 allow the use of doubly-quoted
>>>> strings and single-octet quoting by \.
>>>>
>>>> OPC uses content types as part of [Content_Types].xml
>>>> The XSD schema for this document is opc-contentTypes.xsd.
>>>> It has an ugly regular expression
>>>>
>>>>
>>>> "(((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@
>>>> ,;:\\"/\[\]\?=\{\}\s\t]])+))/((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@
>>>> ,;:\\"/\[\]\?=\{\}\s\t]])+))((\s+)*;(\s+)*(((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@
>>>> ,;:\\"/\[\]\?=\{\}\s\t]])+))=((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@
>>>> ,;:\\"/\[\]\?=\{\}\s\t]])+)|("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"))))*)"
>>>>
>>>> It is not at all clear whether this is equivalent to RFC 2616,
>>>> especially because XML has its own mechanism for character
>>>> escaping (&#x) and also because double quotation marks
>>>> cannot be used within doubly-quoted attribute values.
>>>>
>>>> I tried to reformulate the above regular expression. First,
>>>>
>>>> [\p{IsBasicLatin}-[\p{Cc}\(\)<>@
>>>> ,;:\\"/\[\]\?=\{\}\s\t]]
>>>>
>>>> appears repeatedly. If we represent this string by an internal
>>>> text entity X by introducing
>>>>
>>>> <!ENTITY X "[\p{IsBasicLatin}-[\p{Cc}\(\)<>@
>>>> ,;:\\"/\[\]\?=\{\}\s\t]]">
>>>>
>>>> the entire expression will become
>>>>
>>>>
>>>> "(((($X)+))/((($X)+))((\s+)*;(\s+)*(((($X)+))=((($X)+)|("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"))))*)"
>>>>
>>>>
>>>> By removing unnecessary parentheses, this can be rewritten as
>>>>
>>>> "$X+/$X+(\s*;\s*
>>>> ($X+=(($X+)|("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"))))*"
>>>>
>>>> This looks similar to what RFC 2616 defines. But
>>>> are they equivalent?
>>>>
>>>> Regards,
>>>> Makoto
>>>>
>>>>
>>>>
>>>> 2014-10-21 6:04 GMT+09:00 Arms, Caroline <caar at loc.gov>:
>>>>
>>>> All,
>>>>>
>>>>> I started back on the Content type vs. Media type issue and ran into
>>>>> the question of which RFC(s) we should refer to, thinking that would be a
>>>>> good place to start thinking about rewording things. It's not so simple!
>>>>>
>>>>> Part 2 currently refers to RFC 2616, which may not have been the most
>>>>> appropriate RFC but that is now moot, because 2616 is obsolete and has been
>>>>> replaced by a group of RFCs including RFC 7231 which refers to RFC 2046 in
>>>>> its Media Type subclause but does not elaborate on what media-type actually
>>>>> is. RFC 7231 provides ABNF for media-type, but you need to refer to RFC
>>>>> 7230 for an explanation of "OWS" -- used in the ABNF. RFC 2046 lists the
>>>>> top-level media types and common subtypes. It discusses parameters. Its
>>>>> introduction refers to RFC 2045 for the Content Type context and to RFC 822
>>>>> for all relevant ABNF not found in its Appendix A: Collected Grammar.
>>>>> Media-type is not mentioned in Appendix A. RFC 2045 has a copy of the
>>>>> relevant ABNF from RFC 822.
>>>>>
>>>>> More detailed detective work with URLs is attached below.
>>>>>
>>>>> The question will be how best to refer to this in Part 2. RFC 7231
>>>>> is most convenient for getting the ABNF syntax, but you need RFC 2046 to
>>>>> understand the semantics.
>>>>>
>>>>> To be continued, no doubt ...
>>>>>
>>>>> Caroline
>>>>>
>>>>> Caroline Arms
>>>>> Library of Congress Contractor
>>>>> Co-compiler of Sustainability of Digital Formats resource
>>>>> http://www.digitalpreservation.gov/formats/
>>>>>
>>>>> ** Views expressed are personal and not necessarily those of the
>>>>> institution **
>>>>>
>>>>> ==== DETAILED detective work ====
>>>>>
>>>>> Part 2 currently refers to RFC 2616
>>>>>
>>>>> https://www.mnot.net/blog/2014/06/07/rfc2616_is_dead
>>>>>
>>>>> http://www.rfc-editor.org/info/rfc2616 is marked as obsolete
>>>>>
>>>>> So I went to one of the replacement RFCs
>>>>>
>>>>> http://tools.ietf.org/html/rfc7231
>>>>>
>>>>> 3.1.1.1. Media Type
>>>>>
>>>>> HTTP uses Internet media types [RFC2046] in the Content-Type
>>>>> (Section 3.1.1.5) and Accept (Section 5.3.2) header fields in order
>>>>> to provide open and extensible data typing and type negotiation.
>>>>> Media types define both a data format and various processing models:
>>>>> how to process that data in accordance with each context in which it
>>>>> is received.
>>>>>
>>>>> media-type = type "/" subtype *( OWS ";" OWS parameter )
>>>>> type = token
>>>>> subtype = token
>>>>>
>>>>> The type/subtype MAY be followed by parameters in the form of
>>>>> name=value pairs.
>>>>>
>>>>> parameter = token "=" ( token / quoted-string )
>>>>>
>>>>> The type, subtype, and parameter name tokens are case-insensitive.
>>>>> Parameter values might or might not be case-sensitive, depending on
>>>>> the semantics of the parameter name. The presence or absence of a
>>>>> parameter might be significant to the processing of a media-type,
>>>>> depending on its definition within the media type registry.
>>>>>
>>>>> A parameter value that matches the token production can be
>>>>> transmitted either as a token or within a quoted-string. The quoted
>>>>> and unquoted values are equivalent. For example, the following
>>>>> examples are all equivalent, but the first is preferred for
>>>>> consistency:
>>>>>
>>>>> text/html;charset=utf-8
>>>>> text/html;charset=UTF-8
>>>>> Text/HTML;Charset="utf-8"
>>>>> text/html; charset="utf-8"
>>>>>
>>>>> Internet media types ought to be registered with IANA according to
>>>>> the procedures defined in [BCP13].
>>>>>
>>>>> Note: Unlike some similar constructs in other header fields,
>>>>> media
>>>>> type parameters do not allow whitespace (even "bad" whitespace)
>>>>> around the "=" character.
>>>>>
>>>>> === aside on OWS -- optional whitespace ===
>>>>>
>>>>> OWS = <OWS, see [RFC7230], Section 3.2.3>
>>>>>
>>>>> http://tools.ietf.org/html/rfc7230#section-3.2.3
>>>>>
>>>>> 3.2.3. Whitespace
>>>>>
>>>>> This specification uses three rules to denote the use of linear
>>>>> whitespace: OWS (optional whitespace), RWS (required whitespace),
>>>>> and
>>>>> BWS ("bad" whitespace).
>>>>>
>>>>> The OWS rule is used where zero or more linear whitespace octets
>>>>> might appear. For protocol elements where optional whitespace is
>>>>> preferred to improve readability, a sender SHOULD generate the
>>>>> optional whitespace as a single SP; otherwise, a sender SHOULD NOT
>>>>> generate optional whitespace except as needed to white out invalid
>>>>> or
>>>>> unwanted protocol elements during in-place message filtering.
>>>>>
>>>>> The RWS rule is used when at least one linear whitespace octet is
>>>>> required to separate field tokens. A sender SHOULD generate RWS as
>>>>> a
>>>>> single SP.
>>>>>
>>>>> The BWS rule is used where the grammar allows optional whitespace
>>>>> only for historical reasons. A sender MUST NOT generate BWS in
>>>>> messages. A recipient MUST parse for such bad whitespace and remove
>>>>> it before interpreting the protocol element.
>>>>>
>>>>> OWS = *( SP / HTAB )
>>>>> ; optional whitespace
>>>>> RWS = 1*( SP / HTAB )
>>>>> ; required whitespace
>>>>> BWS = OWS
>>>>> ; "bad" whitespace
>>>>>
>>>>> ==== end of OWS digression
>>>>>
>>>>>
>>>>> http://tools.ietf.org/html/rfc2046
>>>>>
>>>>> Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types
>>>>>
>>>>> Introduction
>>>>>
>>>>> The first document in this set, RFC 2045, defines a number of header
>>>>> fields, including Content-Type. The Content-Type field is used to
>>>>> specify the nature of the data in the body of a MIME entity, by
>>>>> giving media type and subtype identifiers, and by providing
>>>>> auxiliary
>>>>> information that may be required for certain media types. After the
>>>>> type and subtype names, the remainder of the header field is simply
>>>>> a
>>>>> set of parameters, specified in an attribute/value notation. The
>>>>> ordering of parameters is not significant.
>>>>>
>>>>> In general, the top-level media type is used to declare the general
>>>>> type of data, while the subtype specifies a specific format for that
>>>>> type of data. Thus, a media type of "image/xyz" is enough to tell a
>>>>> user agent that the data is an image, even if the user agent has no
>>>>> knowledge of the specific image format "xyz". Such information can
>>>>> be used, for example, to decide whether or not to show a user the
>>>>> raw
>>>>> data from an unrecognized subtype -- such an action might be
>>>>> reasonable for unrecognized subtypes of "text", but not for
>>>>> unrecognized subtypes of "image" or "audio". For this reason,
>>>>> registered subtypes of "text", "image", "audio", and "video" should
>>>>> not contain embedded information that is really of a different type.
>>>>> Such compound formats should be represented using the "multipart" or
>>>>> "application" types.
>>>>>
>>>>> Parameters are modifiers of the media subtype, and as such do not
>>>>> fundamentally affect the nature of the content. The set of
>>>>> meaningful parameters depends on the media type and subtype. Most
>>>>> parameters are associated with a single specific subtype. However,
>>>>> a
>>>>> given top-level media type may define parameters which are
>>>>> applicable
>>>>> to any subtype of that type. Parameters may be required by their
>>>>> defining media type or subtype or they may be optional. MIME
>>>>> implementations must also ignore any parameters whose names they do
>>>>> not recognize.
>>>>>
>>>>> RFC 2046 lists the top-level media types and their subtypes. As shown
>>>>> in the excerpt above, it refers to RFC 2045 for the Content Type header
>>>>> field in the Introduction.
>>>>>
>>>>> ABNF for media type is not defined in RFC 2046 but is defined in RFC
>>>>> 2045 which copies it from RFC 822. RFC 2046 has a Collected Grammar
>>>>> appendix which refers to RFC 822.
>>>>>
>>>>> http://tools.ietf.org/html/rfc2045#page-12
>>>>>
>>>>> 5.1. Syntax of the Content-Type Header Field
>>>>>
>>>>> In the Augmented BNF notation of RFC 822, a Content-Type header
>>>>> field
>>>>> value is defined as follows:
>>>>>
>>>>> content := "Content-Type" ":" type "/" subtype
>>>>> *(";" parameter)
>>>>> ; Matching of media type and subtype
>>>>> ; is ALWAYS case-insensitive.
>>>>>
>>>>> type := discrete-type / composite-type
>>>>>
>>>>> discrete-type := "text" / "image" / "audio" / "video" /
>>>>> "application" / extension-token
>>>>>
>>>>> composite-type := "message" / "multipart" / extension-token
>>>>>
>>>>> extension-token := ietf-token / x-token
>>>>>
>>>>> ietf-token := <An extension token defined by a
>>>>> standards-track RFC and registered
>>>>> with IANA.>
>>>>>
>>>>> x-token := <The two characters "X-" or "x-" followed, with
>>>>> no intervening white space, by any token>
>>>>>
>>>>> subtype := extension-token / iana-token
>>>>>
>>>>> iana-token := <A publicly-defined extension token. Tokens
>>>>> of this form must be registered with IANA
>>>>> as specified in RFC 2048.>
>>>>>
>>>>> parameter := attribute "=" value
>>>>>
>>>>> attribute := token
>>>>> ; Matching of attributes
>>>>> ; is ALWAYS case-insensitive.
>>>>>
>>>>> value := token / quoted-string
>>>>>
>>>>> token := 1*<any (US-ASCII) CHAR except SPACE, CTLs,
>>>>> or tspecials>
>>>>>
>>>>> tspecials := "(" / ")" / "<" / ">" / "@" /
>>>>> "," / ";" / ":" / "\" / <">
>>>>> "/" / "[" / "]" / "?" / "="
>>>>> ; Must be in quoted-string,
>>>>> ; to use within parameter values
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Praying for the victims of the Japan Tohoku earthquake
>>>>
>>>> Makoto
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Praying for the victims of the Japan Tohoku earthquake
>>>
>>> Makoto
>>>
>>
>>
>>
>> --
>>
>> Praying for the victims of the Japan Tohoku earthquake
>>
>> Makoto
>>
>
>
>
> --
>
> Praying for the victims of the Japan Tohoku earthquake
>
> Makoto
>
--
Praying for the victims of the Japan Tohoku earthquake
Makoto
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.vse.cz/pipermail/sc34wg4/attachments/20141211/26992b2e/attachment-0001.html>
More information about the sc34wg4
mailing list