From 165e2cb8c917c570bfb7cb98155588ccdddb154a Mon Sep 17 00:00:00 2001 From: Ashish Kulkarni Date: Fri, 17 Feb 2017 14:54:22 +0530 Subject: [PATCH] document issue with FormRequest.from_response due to bug in lxml This can make the spider fail due to incorrect values being posted server-side, which is extremely hard to debug because it is easy to miss leading/trailing whitespace, even with a logging proxy. The fix was merged for lxml 3.8 in lxml/lxml#228 so document that as well. --- docs/topics/request-response.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index 1fdd26043..a003026bb 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -376,6 +376,10 @@ fields with form data from :class:`Response` objects. control clicked (instead of disabling it) you can also use the ``clickdata`` argument. + .. caution:: Using this method with select elements which have leading + or trailing whitespace in the option values will not work due to a + `bug in lxml`_, which should be fixed in lxml 3.8 and above. + :param response: the response containing a HTML form which will be used to pre-populate the form fields :type response: :class:`Response` object @@ -710,3 +714,4 @@ XmlResponse objects line. See :attr:`TextResponse.encoding`. .. _Twisted Failure: https://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html +.. _bug in lxml: https://bugs.launchpad.net/lxml/+bug/1665241