<div dir="ltr"><div>Folks,<br></div><div><br></div><div>The regular expression in opc-contentTypes.xsd has another interesting</div><div>subexpression:</div><div><br></div><div>("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"</div><div><br></div><div>This subexpression matches a doubly-quoted string. But what character</div><div>is allowed as a part of this doubly-quoted string? Is this </div><div>subexpression consistent with RFC 7230?</div><div><br></div><div>First, unlike the first regular expression, </div><div>neither \(\)<>@,;:\\ nor /\[\]\?=\{\}\s\t are excluded. </div><div>This is because RFCs 2045 and 7230 allow tspecials as part of </div><div>doubly-quoted strings. This is nice.</div><div><br></div><div>Second, escaped characters such as \a are always allows by</div><div>(\\[\p{IsBasicLatin}]) as long as escaped characters are from #x0000</div><div>to #x007F. RFC 7230 allows more characters (#x0080 to #x007F) to be</div><div>escaped, but does not allow invisible characters such as the space</div><div>character to be escaped. Thus, there is a discrepancy here.</div><div><br></div><div>Third, \p{IsLatin-1Supplement} represents characters </div><div>from #x0080 to #x00FF. See <a href="http://www.w3.org/TR/xmlschema-2/#nt-charClassEsc">http://www.w3.org/TR/xmlschema-2/#nt-charClassEsc</a></div><div>This matches obs-text in RFC 7230.</div><div><br></div><div>Fourth, RFC 7230 (to be precise, qdtext) does not allow REVERSE</div><div>SOLIDUS, but the subexpression does. Thus, we have another </div><div>discrepancy.</div><div><br></div><div> qdtext = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text</div><div><br></div><div>Regards,</div><div>Makoto</div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-11-02 16:56 GMT+09:00 MURATA Makoto <span dir="ltr"><<a href="mailto:eb2m-mrt@asahi-net.or.jp" target="_blank">eb2m-mrt@asahi-net.or.jp</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Oops, I forgot to point out that <span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">\{ and \} are disallowed by </span><div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">our regular expression, but they are not tspecials as </span></div><div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:14px">specified in RFC </span><span style="font-family:arial,sans-serif;font-size:14px">2045. </span><span style="font-family:arial,sans-serif;font-size:14px">RFC 7230 does </span></div><div><span style="font-family:arial,sans-serif;font-size:14px">not allow these two characters, and exactly catches </span></div><div><span style="font-family:arial,sans-serif;font-size:14px">the enumerated list in my previous mail.</span></div><div><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div><div><font face="arial, sans-serif"><span style="font-size:14px"><a href="https://tools.ietf.org/html/rfc7230#section-3.2.6" target="_blank">https://tools.ietf.org/html/rfc7230#section-3.2.6</a></span></font><br></div><div><font face="arial, sans-serif"><span style="font-size:14px"><br></span></font></div><div><font face="arial, sans-serif"><span style="font-size:14px">One could say that our regular expression is already </span></font></div><div><font face="arial, sans-serif"><span style="font-size:14px">aligned with RFC 7230 rather than RFC 2045.</span></font></div><div><font face="arial, sans-serif"><span style="font-size:14px"><br></span></font></div><div><font face="arial, sans-serif"><span style="font-size:14px">Regards,</span></font></div><div><font face="arial, sans-serif"><span style="font-size:14px">Makoto</span></font></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2014-11-02 16:15 GMT+09:00 MURATA Makoto <span dir="ltr"><<a href="mailto:eb2m-mrt@asahi-net.or.jp" target="_blank">eb2m-mrt@asahi-net.or.jp</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Folks,<div><br></div><div>I pointed out that<span><div><br></div><div> <span style="font-family:arial,sans-serif;font-size:14px">[\p{IsBasicLatin}-[\p{Cc}&#</span><span style="font-family:arial,sans-serif;font-size:14px">127;\(\)<>@,;:\\"/\</span><span style="font-family:arial,sans-serif;font-size:14px">[\]\?=\{\}\s\t]] </span></div><div><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div></span><div><span style="font-family:arial,sans-serif;font-size:14px">is </span><span style="font-family:arial,sans-serif;font-size:14px">used repeatedly. This appears to represent characters in </span><div><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div><div><span><span style="font-family:arial,sans-serif;font-size:14px"> token := 1*<any (US-ASCII) CHAR except SPACE, CTLs,</span><br style="font-family:arial,sans-serif;font-size:14px"><span style="font-family:arial,sans-serif;font-size:14px"> or tspecials></span><br style="font-family:arial,sans-serif;font-size:14px"><br></span>where</div><span><br style="font-family:arial,sans-serif;font-size:14px"><br style="font-family:arial,sans-serif;font-size:14px"><span style="font-family:arial,sans-serif;font-size:14px"> tspecials := "(" / ")" / "<" / ">" / "@" /</span><br style="font-family:arial,sans-serif;font-size:14px"><span style="font-family:arial,sans-serif;font-size:14px"> "," / ";" / ":" / "\" / <"></span><br style="font-family:arial,sans-serif;font-size:14px"><span style="font-family:arial,sans-serif;font-size:14px"> "/" / "[" / "]" / "?" / "="</span><br style="font-family:arial,sans-serif;font-size:14px"><span style="font-family:arial,sans-serif;font-size:14px"> ; Must be in quoted-string,</span><br style="font-family:arial,sans-serif;font-size:14px"></span><div><span style="font-family:arial,sans-serif;font-size:14px"> ; to use within parameter values</span><span style="font-family:arial,sans-serif;font-size:14px">" /</span><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div></div><div><span style="font-family:arial,sans-serif;font-size:14px"><br></span></div><div><font face="arial, sans-serif"><span style="font-size:14px">They both represent </span></font><span style="font-family:arial,sans-serif;font-size:14px">any of the following characters:</span></div><div><font face="arial, sans-serif"><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">- 0021;EXCLAMATION MARK;Po;0;ON;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 0023;NUMBER SIGN;Po;0;ET;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 0024;DOLLAR SIGN;Sc;0;ET;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 0025;PERCENT SIGN;Po;0;ET;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 0026;AMPERSAND;Po;0;ON;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 0027;APOSTROPHE;Po;0;ON;;;;;N;APOSTROPHE-QUOTE;;;;</span></div><div><span style="font-size:14px">- 002A;ASTERISK;Po;0;ON;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 002B;PLUS SIGN;Sm;0;ES;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 002D;HYPHEN-MINUS;Pd;0;ES;;;;;N;;;;;</span></div><div><span style="font-size:14px">- 002E;FULL STOP;Po;0;CS;;;;;N;PERIOD;;;;</span></div><div><span style="font-size:14px">- 0-9</span></div><div><span style="font-size:14px">- A-Z</span></div><div><span style="font-size:14px">- 005E;CIRCUMFLEX ACCENT;Sk;0;ON;;;;;N;SPACING CIRCUMFLEX;;;;</span></div><div><span style="font-size:14px">- 005F;LOW LINE;Pc;0;ON;;;;;N;SPACING UNDERSCORE;;;;</span></div><div><span style="font-size:14px">- 0060;GRAVE ACCENT;Sk;0;ON;;;;;N;SPACING GRAVE;;;;</span></div><div><span style="font-size:14px">- a-z</span></div><div><span style="font-size:14px">- 007C;VERTICAL LINE;Sm;0;ON;;;;;N;VERTICAL BAR;;;;</span></div><div><span style="font-size:14px">- 007E;TILDE;Sm;0;ON;;;;;N;;;;;</span></div><div style="font-size:14px"><br></div><div style="font-size:14px"><br></div><div style="font-size:14px"><div>The regular expression allow any of these characters as </div><div>part of a top-level media type name, second-level </div><div>media type name, and parameter name,</div><div><br></div><div>Regards,</div><div>Makoto</div></div></font></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">2014-10-25 22:08 GMT+09:00 MURATA Makoto <span dir="ltr"><<a href="mailto:eb2m-mrt@asahi-net.or.jp" target="_blank">eb2m-mrt@asahi-net.or.jp</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Caroline,<div><br></div><div>Thank you for your through study! This is an </div><div>eye opener.</div><div><br></div><div>Both RFC 2616 and RFC 7321 allow the use of doubly-quoted </div><div>strings and <span style="color:rgb(0,0,0);font-size:1em">single-octet quoting by \.</span></div><div><span style="color:rgb(0,0,0);font-size:1em"><br></span></div><div><span style="color:rgb(0,0,0);font-size:1em">OPC uses content types as part of </span>[Content_Types].xml </div><div>The XSD schema for this document is opc-contentTypes.xsd.</div><div>It has an ugly regular expression </div><div><table><tbody><tr><td><br></td><td><br></td></tr></tbody></table><div>"(((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t]])+))/((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t]])+))((\s+)*;(\s+)*(((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t]])+))=((([\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t]])+)|("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"))))*)"</div><div><br></div><div>It is not at all clear whether this is equivalent to RFC 2616, </div><div>especially because XML has its own mechanism for character </div><div>escaping (&#x) and also because double quotation marks </div><div>cannot be used within doubly-quoted attribute values.</div><div><br></div><div>I tried to reformulate the above regular expression. First, </div><div><br></div><div>[\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t]]<br></div><div><br></div><div>appears repeatedly. If we represent this string by an internal </div><div>text entity X by introducing </div><div><br></div><div><!ENTITY X "[\p{IsBasicLatin}-[\p{Cc}\(\)<>@,;:\\"/\[\]\?=\{\}\s\t]]"></div><div><br></div><div>the entire expression will become</div><div><br></div><div>"(((($X)+))/((($X)+))((\s+)*;(\s+)*(((($X)+))=((($X)+)|("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"))))*)"<br></div><div><br></div><div><br></div><div>By removing unnecessary parentheses, this can be rewritten as</div><div><br></div><div>"$X+/$X+(\s*;\s* ($X+=(($X+)|("(([\p{IsLatin-1Supplement}\p{IsBasicLatin}-[\p{Cc}"\n\r]]|(\s+))|(\\[\p{IsBasicLatin}]))*"))))*"<br></div><div><br></div><div>This looks similar to what RFC 2616 defines. But </div><div>are they equivalent?</div><div><br></div><div>Regards,</div><div>Makoto</div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-10-21 6:04 GMT+09:00 Arms, Caroline <span dir="ltr"><<a href="mailto:caar@loc.gov" target="_blank">caar@loc.gov</a>></span>:<div><div><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">All,<br>
<br>
I started back on the Content type vs. Media type issue and ran into the question of which RFC(s) we should refer to, thinking that would be a good place to start thinking about rewording things. It's not so simple!<br>
<br>
Part 2 currently refers to RFC 2616, which may not have been the most appropriate RFC but that is now moot, because 2616 is obsolete and has been replaced by a group of RFCs including RFC 7231 which refers to RFC 2046 in its Media Type subclause but does not elaborate on what media-type actually is. RFC 7231 provides ABNF for media-type, but you need to refer to RFC 7230 for an explanation of "OWS" -- used in the ABNF. RFC 2046 lists the top-level media types and common subtypes. It discusses parameters. Its introduction refers to RFC 2045 for the Content Type context and to RFC 822 for all relevant ABNF not found in its Appendix A: Collected Grammar. Media-type is not mentioned in Appendix A. RFC 2045 has a copy of the relevant ABNF from RFC 822.<br>
<br>
More detailed detective work with URLs is attached below.<br>
<br>
The question will be how best to refer to this in Part 2. RFC 7231 is most convenient for getting the ABNF syntax, but you need RFC 2046 to understand the semantics.<br>
<br>
To be continued, no doubt ...<br>
<br>
Caroline<br>
<br>
Caroline Arms<br>
Library of Congress Contractor<br>
Co-compiler of Sustainability of Digital Formats resource <a href="http://www.digitalpreservation.gov/formats/" target="_blank">http://www.digitalpreservation.gov/formats/</a><br>
<br>
** Views expressed are personal and not necessarily those of the institution **<br>
<br>
==== DETAILED detective work ====<br>
<br>
Part 2 currently refers to RFC 2616<br>
<br>
<a href="https://www.mnot.net/blog/2014/06/07/rfc2616_is_dead" target="_blank">https://www.mnot.net/blog/2014/06/07/rfc2616_is_dead</a><br>
<br>
<a href="http://www.rfc-editor.org/info/rfc2616" target="_blank">http://www.rfc-editor.org/info/rfc2616</a> is marked as obsolete<br>
<br>
So I went to one of the replacement RFCs<br>
<br>
<a href="http://tools.ietf.org/html/rfc7231" target="_blank">http://tools.ietf.org/html/rfc7231</a><br>
<br>
3.1.1.1. Media Type<br>
<br>
HTTP uses Internet media types [RFC2046] in the Content-Type<br>
(Section 3.1.1.5) and Accept (Section 5.3.2) header fields in order<br>
to provide open and extensible data typing and type negotiation.<br>
Media types define both a data format and various processing models:<br>
how to process that data in accordance with each context in which it<br>
is received.<br>
<br>
media-type = type "/" subtype *( OWS ";" OWS parameter )<br>
type = token<br>
subtype = token<br>
<br>
The type/subtype MAY be followed by parameters in the form of<br>
name=value pairs.<br>
<br>
parameter = token "=" ( token / quoted-string )<br>
<br>
The type, subtype, and parameter name tokens are case-insensitive.<br>
Parameter values might or might not be case-sensitive, depending on<br>
the semantics of the parameter name. The presence or absence of a<br>
parameter might be significant to the processing of a media-type,<br>
depending on its definition within the media type registry.<br>
<br>
A parameter value that matches the token production can be<br>
transmitted either as a token or within a quoted-string. The quoted<br>
and unquoted values are equivalent. For example, the following<br>
examples are all equivalent, but the first is preferred for<br>
consistency:<br>
<br>
text/html;charset=utf-8<br>
text/html;charset=UTF-8<br>
Text/HTML;Charset="utf-8"<br>
text/html; charset="utf-8"<br>
<br>
Internet media types ought to be registered with IANA according to<br>
the procedures defined in [BCP13].<br>
<br>
Note: Unlike some similar constructs in other header fields, media<br>
type parameters do not allow whitespace (even "bad" whitespace)<br>
around the "=" character.<br>
<br>
=== aside on OWS -- optional whitespace ===<br>
<br>
OWS = <OWS, see [RFC7230], Section 3.2.3><br>
<br>
<a href="http://tools.ietf.org/html/rfc7230#section-3.2.3" target="_blank">http://tools.ietf.org/html/rfc7230#section-3.2.3</a><br>
<br>
3.2.3. Whitespace<br>
<br>
This specification uses three rules to denote the use of linear<br>
whitespace: OWS (optional whitespace), RWS (required whitespace), and<br>
BWS ("bad" whitespace).<br>
<br>
The OWS rule is used where zero or more linear whitespace octets<br>
might appear. For protocol elements where optional whitespace is<br>
preferred to improve readability, a sender SHOULD generate the<br>
optional whitespace as a single SP; otherwise, a sender SHOULD NOT<br>
generate optional whitespace except as needed to white out invalid or<br>
unwanted protocol elements during in-place message filtering.<br>
<br>
The RWS rule is used when at least one linear whitespace octet is<br>
required to separate field tokens. A sender SHOULD generate RWS as a<br>
single SP.<br>
<br>
The BWS rule is used where the grammar allows optional whitespace<br>
only for historical reasons. A sender MUST NOT generate BWS in<br>
messages. A recipient MUST parse for such bad whitespace and remove<br>
it before interpreting the protocol element.<br>
<br>
OWS = *( SP / HTAB )<br>
; optional whitespace<br>
RWS = 1*( SP / HTAB )<br>
; required whitespace<br>
BWS = OWS<br>
; "bad" whitespace<br>
<br>
==== end of OWS digression<br>
<br>
<br>
<a href="http://tools.ietf.org/html/rfc2046" target="_blank">http://tools.ietf.org/html/rfc2046</a><br>
<br>
Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types<br>
<br>
Introduction<br>
<br>
The first document in this set, RFC 2045, defines a number of header<br>
fields, including Content-Type. The Content-Type field is used to<br>
specify the nature of the data in the body of a MIME entity, by<br>
giving media type and subtype identifiers, and by providing auxiliary<br>
information that may be required for certain media types. After the<br>
type and subtype names, the remainder of the header field is simply a<br>
set of parameters, specified in an attribute/value notation. The<br>
ordering of parameters is not significant.<br>
<br>
In general, the top-level media type is used to declare the general<br>
type of data, while the subtype specifies a specific format for that<br>
type of data. Thus, a media type of "image/xyz" is enough to tell a<br>
user agent that the data is an image, even if the user agent has no<br>
knowledge of the specific image format "xyz". Such information can<br>
be used, for example, to decide whether or not to show a user the raw<br>
data from an unrecognized subtype -- such an action might be<br>
reasonable for unrecognized subtypes of "text", but not for<br>
unrecognized subtypes of "image" or "audio". For this reason,<br>
registered subtypes of "text", "image", "audio", and "video" should<br>
not contain embedded information that is really of a different type.<br>
Such compound formats should be represented using the "multipart" or<br>
"application" types.<br>
<br>
Parameters are modifiers of the media subtype, and as such do not<br>
fundamentally affect the nature of the content. The set of<br>
meaningful parameters depends on the media type and subtype. Most<br>
parameters are associated with a single specific subtype. However, a<br>
given top-level media type may define parameters which are applicable<br>
to any subtype of that type. Parameters may be required by their<br>
defining media type or subtype or they may be optional. MIME<br>
implementations must also ignore any parameters whose names they do<br>
not recognize.<br>
<br>
RFC 2046 lists the top-level media types and their subtypes. As shown in the excerpt above, it refers to RFC 2045 for the Content Type header field in the Introduction.<br>
<br>
ABNF for media type is not defined in RFC 2046 but is defined in RFC<br>
2045 which copies it from RFC 822. RFC 2046 has a Collected Grammar appendix which refers to RFC 822.<br>
<br>
<a href="http://tools.ietf.org/html/rfc2045#page-12" target="_blank">http://tools.ietf.org/html/rfc2045#page-12</a><br>
<br>
5.1. Syntax of the Content-Type Header Field<br>
<br>
In the Augmented BNF notation of RFC 822, a Content-Type header field<br>
value is defined as follows:<br>
<br>
content := "Content-Type" ":" type "/" subtype<br>
*(";" parameter)<br>
; Matching of media type and subtype<br>
; is ALWAYS case-insensitive.<br>
<br>
type := discrete-type / composite-type<br>
<br>
discrete-type := "text" / "image" / "audio" / "video" /<br>
"application" / extension-token<br>
<br>
composite-type := "message" / "multipart" / extension-token<br>
<br>
extension-token := ietf-token / x-token<br>
<br>
ietf-token := <An extension token defined by a<br>
standards-track RFC and registered<br>
with IANA.><br>
<br>
x-token := <The two characters "X-" or "x-" followed, with<br>
no intervening white space, by any token><br>
<br>
subtype := extension-token / iana-token<br>
<br>
iana-token := <A publicly-defined extension token. Tokens<br>
of this form must be registered with IANA<br>
as specified in RFC 2048.><br>
<br>
parameter := attribute "=" value<br>
<br>
attribute := token<br>
; Matching of attributes<br>
; is ALWAYS case-insensitive.<br>
<br>
value := token / quoted-string<br>
<br>
token := 1*<any (US-ASCII) CHAR except SPACE, CTLs,<br>
or tspecials><br>
<br>
tspecials := "(" / ")" / "<" / ">" / "@" /<br>
"," / ";" / ":" / "\" / <"><br>
"/" / "[" / "]" / "?" / "="<br>
; Must be in quoted-string,<br>
; to use within parameter values<br>
</blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><br>Praying for the victims of the Japan Tohoku earthquake<br><br>Makoto
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><br>Praying for the victims of the Japan Tohoku earthquake<br><br>Makoto</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><br>Praying for the victims of the Japan Tohoku earthquake<br><br>Makoto</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><br>Praying for the victims of the Japan Tohoku earthquake<br><br>Makoto</div>
</div>