Look forward to is one of the many phrasal verbs in English in which an adverbial particle (forward) as well as a preposition (to) is combined with the stem verb to signify a particular meaning. What we are looking forward to can be exemplified as either as a noun phrase or as a verb-phrase with an -ing pattern

• Jill says she's not looking forward to Jack's party next weekend.
  
• I very much look forward to meeting you soon.
  
• They're looking forward to joining their children in Australia

There are many such three-part verbs, e.g.:

look back on = think back to
put up with = tolerate
come down with = fall ill with

There are a number of instances where such verbs end with the preposition to, e.g.:

face up to = confront
get round to = do something after some delay
get down to = concentrate on

Note that in such instances to is not part of any infinitive phrase. It is an integral part of the verb. And whatever it is that we face up to or get round to is normally expressed as either a noun phrase or as a verb phrase with an -ing pattern:

• I must get round to cleaning my car next weekend.
  
• And I must get down to reading Jack's article which he sent me two weeks ago
  
• I must face up to the fact that I'm never going to be promoted in this organisation.

Note that when verbs follow prepositions (any prepositions) the V-ing form is normally used, not the to-infinitive pattern:

• I managed to finish reading Jack's article by staying up till midnight.
  
• He's talking about getting it published in National Geographic magazine.
  
• Instead of going on holiday last summer, he undertook this arduous trip up the Amazon.
  

agree - agree to

There is a complication in your example, Jolie, where both the -ing form and the to-inifnitive pattern appear possible:

• I cannot agree to share / to sharing an office with Ben.
  
• In no way can I agree to sharing / to share an office with Ben

The complication arises because there are two different forms of pretty much the same verb, agree and agree to. If we are using the phrasal verb, agree to, the -ing pattern is more likely. If we are using the non-phrasal verb, agree, the to-infinitive pattern is imperative. Compare the following:

• What have you agreed?
  We've agreed to tidy our rooms when we get up, to clear the dishes from the table after eating and not to go out until we've finished our homework.
  
  
• What have you agreed to?
  We've agreed to arriving punctually before the working day begins and to not leaving before five o' clock in the afternoon.
  

Note that the opposite of agree to is object to and here only the -ing pattern is possible:

• What do you object to in her behaviour?
  I object to her going out every evening and not telling me where she is going.